AWS Messaging & Streaming Services

Messaging Systems

What is Messaging?

Messaging is a technology that enables high-speed, asynchronous, program-to-program communication with reliable delivery.

How messaging systems work?

  1. Create — The sender creates the message and populates it with data.
  2. Send — The sender adds the message to a channel.
  3. Deliver — The messaging system moves the message from the sender’s computer to the receiver’s computer, making it available to the receiver.
  4. Receive — The receiver reads the message from the channel.
  5. Process — The receiver extracts the data from the message.

What are the advantages of messaging systems?

  1. Send and forget — The sending application sends the message to the message channel. Once that send is complete, the sender can go on to other work while the messaging system transmits the message in the background. The sender can be confident that the receiver will eventually receive the message and does not have to wait until that happens.
  2. Asynchronous communication: Once the message is stored, the sender is then free to perform other work while the message is transmitted in the background

Event Streaming

Message vs Event

  1. What is an event? An event encapsulates a change in state (what has happened).
  2. What is a message ? A message encapsulates the intention / action (what has to happen)
  3. How are events distributed? Streaming and Messaging systems
  4. How are messages distributed? Messaging systems

Reference: https://robertleggett.blog/2020/03/02/choosing-event-streaming-or-messaging-for-your-architecture/

What is a stream?

  1. A stream consists of immutable data, only inserting new events, whereas existing events cannot be changed.
  2. Streams are persistent, durable and fault tolerant.

What is streaming?

  1. Streaming of data is the constant flow of events where each event should contain enough information to reflect the change in state.
  2. It allows for the processing of data to occur in real-time (data in motion) and is different from the traditional approach for the processing of static data to occur (data at rest) at a later point in time, known as batch processing
  3. Streaming data is unbounded, meaning it has no real beginning and no real end
  4. Each event is processed as it occurs and is managed accordingly

Reference: https://robertleggett.blog/2020/03/02/choosing-event-streaming-or-messaging-for-your-architecture/

Services in AWS for messaging and streaming
  1. Amazon MQ
    1. Use case: Migrate to a managed message broker to automate software administration and maintenance, without having to rewrite existing applications
  2. SQS
    1. Use case: Build decoupled, highly scalable microservices, distributed systems, and serverless applications in the cloud
  3. SNS
    1. Use case: Push messages to a variety of endpoints and clients in distributed systems, microservices, and serverless applications and enable event-driven architecture
  4. Kinesis Streams
    1. Use case: Build custom, real-time applications that process data streams using popular stream processing frameworks
  5. IOT Message broker
    1. Use case: Send messages to/from devices and AWS IoT apps in a secure fashion using MQTT, HTTP, and WebSockets

SQS
  1. Queue Types :
    1. Standard :
      1. Messages are delivered at least once.
      2. You can use standard message queues in many scenarios, as long as your application can process messages that arrive more than once and out of order
    2. FIFO:
      1. Messages are delivered exactly once.
      2. FIFO queues are designed to enhance messaging between applications when the order of operations and events is critical, or where duplicates can’t be tolerated
  2. Polling
    1. What is polling? A way that one system continuously checks other systems to see what state they are in and if they have any message to communicate.SQS supports two types of polling short polling and long polling
    2. Short polling:
    3. Long polling: When a consumer poll’s the queue and if it doesn’t find a message it could still wait for up to 20 seconds and receive the message instantaneously if a message comes into the queue.
    4. When to use short polling?
      1. If your application expects an immediate response.
      2. Example: If your application uses a single thread to poll multiple queues, switching from short polling to long polling will probably not work, because the single thread will wait for the long-poll timeout on any empty queues, delaying the processing of any queues that might contain messages.
  1. Visibility timeout
    1. When a customer receives a message, it must delete the message from the queue.
    2. But it is not guaranteed that the customer really received the message.
    3. So to prevent other consumers from processing the message again, Amazon SQS sets a visibility timeout, a period of time during which Amazon SQS prevents other consumers from receiving and processing the message.
  2. Dead letter queue: The Dead Letter Queue is a secondary queue that receives messages from the first queue after a certain number of times the message wasn’t processed on the main queue. It’s a way to store problematic messages in a separate queue for further analysis.

Amazon MQ

Many businesses worldwide are using two kinds of message-oriented middleware or message

brokers, 1/ Commercial Brokers such as IBM MQ & TIBCO EMS, and 2/ Open-sourced brokers such as Apache ActiveMQ & RabbitMQ. These businesses often discover some challenges in managing these brokers.

Amazon MQ is a managed message broker service, which makes it easy to set up and operate message brokers in AWS.

  • Advantage :Customers can leverage Amazon MQ to connect new cloud-native applications to their on-premise systems or to migrate one or more existing applications to the cloud while maintaining interoperability with back end systems and standards compliance.
  • Broker engine types
  • Apache ActiveMQ Deployment modes
    1. Single instance
    2. Active / standby
    3. Network of brokers
  • RabbitMQ deployment modes
    1. Single Instance broker
    2. Cluster deployment
  • MQ ENI : When you first create an Amazon MQ broker, Amazon MQ provisions an elastic network interface in the Virtual Private Cloud (VPC) under your account. The network interface allows your client (producer or consumer) to communicate with the Amazon MQ broker.
  • Authentication
    1. Active MQ
      1. Native Active MQ authentication
      2. LDAP Authentication
    2. Rabbit MQ
      1. Native RabbitMQ authentication
  • Encryption
    1. At rest : Data can be encrypted by using security keys stored in KMS
    2. In transit : All connections between Amazon MQ brokers use Transport layer Security (TLS) to provide encryption in transit.
  • SNS

    SNS is a fully managed pub/sub messaging service that lets you fan out messages to large numbers of recipients at one time, using topics.

    1. SNS Topic: An Amazon SNS topic is a logical access point that acts as a communication channel. A topic lets you group multiple endpoints (such as AWS Lambda, Amazon SQS, HTTP/S, or an email address).
    2. SNS Topic FIFO:
      1. Using SNS FIFO topics and SQS FIFO queues enables the processing of messages in order and with no duplication.
    3. Message filtering:
      1. By default, each subscriber receives every message published to the topic.
      2. To receive a subset of the messages, a subscriber must assign a filter policy to the topic subscription.
      3. A filter policy is a simple JSON object containing attributes that define which messages the subscriber receives.
    4. Message security: Server-side encryption protects the contents of messages that are stored in Amazon SNS topics, using encryption keys provided by AWS KMS.
    5. Dead letter queue:
      1. A dead-letter queue associated with an Amazon SNS subscription is an ordinary Amazon SQS queue.
      2. Messages that can’t be delivered due to client errors or server errors are held in the dead-letter queue for further analysis or reprocessing. A dead-letter queue associated with an Amazon SNS subscription is an ordinary Amazon SQS
    Kinesis Data Stream
    1. Stream: A Kinesis data stream is a set of shards. Each shard has a sequence of data records. Each data record has a sequence number that is assigned by Kinesis Data Streams
    2. Shard: Every record you write to the stream ends up in exactly one shard, where it is stored in the same order it was written, until it expires. To decide which shard to put a record to, Kinesis uses a so-called partition key.
      Reference and image credit: https://dev.solita.fi/2020/05/28/kinesis-streams-part-1.html
    3. Producer: A producer is an application that writes data to Amazon Kinesis Data Streams. You can build producers for Kinesis Data Streams using the AWS SDK for Java and the Kinesis Producer Library.
      1. Kinesis Producer Library (KPL) (application)
      2. Amazon Kinesis Data Streams API with the AWS SDK
      3. Kinesis agent: Kinesis Agent is a stand-alone Java software application that offers an easy way to collect and send data to Kinesis Data Streams
    4. Consumer: A consumer is an application that processes all data from a Kinesis data stream
      1. Lambda
      2. Kinesis data analytics
      3. Kinesis data firehose
      4. Kinesis client library
    5. Data protection: Server-side encryption using AWS Key Management Service (AWS KMS) keys makes it easy for you to meet strict data management requirements by encrypting your data at rest within Amazon Kinesis Data Streams.
    IOT Message broker ( IOT Core )

    AWS IoT Core provides the services that connect your IoT devices to the AWS Cloud so that other cloud services and applications can interact with your internet-connected devices.

    The message broker distributes device data to devices that have subscribed to it and to other AWS IoT Core services, such as the Device Shadow service and the rules engine