Amazon SQS: The Message Queue

Amazon Simple Queue Service (SQS) is a managed message queue: producers send messages to the queue, consumers retrieve them via polling. The basic semantics are point-to-point: A message in the queue it is read by one consumer at a time (the first one who takes it "hides" it from the others via the visibility timeout).

Standard Queue vs FIFO Queue

SQS offers two modes:

  • Standard Queue: Unlimited throughput, at-least-once delivery (duplicates possible), best-effort ordering (not guaranteed). For high-volume use cases where duplicates are manageable on the consumer side.
  • FIFO Queue: exactly-once delivery (automatic deduplication via MessageDeduplicationId), guaranteed sorting for message group, throughput limited to 3,000 msg/s per request (with batching). For use cases where ordering and duplicates are critical (e.g. financial transactions). 
// SQS Producer Java con AWS SDK v2
import software.amazon.awssdk.services.sqs.*;
import software.amazon.awssdk.services.sqs.model.*;

public class SqsProducer {

    private final SqsClient sqsClient = SqsClient.builder()
        .region(Region.EU_WEST_1)
        .build();

    // Standard Queue: invio semplice
    public void sendToStandardQueue(String queueUrl, String messageBody) {
        SendMessageResponse response = sqsClient.sendMessage(
            SendMessageRequest.builder()
                .queueUrl(queueUrl)
                .messageBody(messageBody)
                // Delay di consegna (0-900 secondi)
                .delaySeconds(0)
                // Attributi del messaggio per filtraggio (SNS subscription filter)
                .messageAttributes(Map.of(
                    "eventType", MessageAttributeValue.builder()
                        .stringValue("OrdineCreato")
                        .dataType("String")
                        .build()
                ))
                .build()
        );
        System.out.println("Messaggio inviato: " + response.messageId());
    }

    // FIFO Queue: richiede MessageGroupId e MessageDeduplicationId
    public void sendToFifoQueue(String queueUrl, String ordineId, String payload) {
        sqsClient.sendMessage(
            SendMessageRequest.builder()
                .queueUrl(queueUrl)  // url deve terminare in .fifo
                .messageBody(payload)
                // Group: tutti i messaggi dello stesso ordine vengono elaborati in ordine
                .messageGroupId("ordine-" + ordineId)
                // Deduplication: prevenzione duplicati (valido 5 minuti)
                .messageDeduplicationId(ordineId + "-" + System.currentTimeMillis())
                .build()
        );
    }
}

SQS Consumer: Polling and Visibility Timeout

// SQS Consumer con Long Polling e gestione DLQ
public class SqsConsumer {

    private final SqsClient sqsClient = SqsClient.builder()
        .region(Region.EU_WEST_1).build();

    public void poll(String queueUrl) {
        while (true) {
            // Long polling: aspetta fino a 20 secondi per nuovi messaggi
            // Riduce le chiamate vuote (e i costi) rispetto al short polling
            ReceiveMessageResponse response = sqsClient.receiveMessage(
                ReceiveMessageRequest.builder()
                    .queueUrl(queueUrl)
                    .maxNumberOfMessages(10)        // max 10 per chiamata
                    .waitTimeSeconds(20)             // long polling
                    .visibilityTimeout(30)           // 30s per elaborare
                    .messageAttributeNames("All")
                    .build()
            );

            for (Message message : response.messages()) {
                try {
                    processMessage(message.body());

                    // Successo: elimina dalla coda
                    sqsClient.deleteMessage(
                        DeleteMessageRequest.builder()
                            .queueUrl(queueUrl)
                            .receiptHandle(message.receiptHandle())
                            .build()
                    );

                } catch (Exception e) {
                    // Non eliminare: SQS rirenderà visibile il messaggio
                    // dopo il visibility timeout. Dopo maxReceiveCount tentativi
                    // finisce nella DLQ configurata
                    System.err.println("Errore, il messaggio tornera visibile: " + e.getMessage());
                }
            }
        }
    }
}

Amazon SNS: Fan-Out Pub/Sub

Amazon Simple Notification Service (SNS) implements the pattern publish/subscribe: a producer posts a message to a SNS topics, and SNS delivers it in parallel to everyone the members (subscriber). An SNS topic can have thousands of subscribers: Lambda, SQS, HTTP endpoint, email, SMS, mobile push.

The SNS + SQS pattern (SNS Fan Out) is one of the most common patterns in AWS architectures: SNS delivers the same event to multiple SQS queues in parallel, and each queue serves a different consumer (a different microservice). 

# SNS + SQS Fan-Out: un evento raggiunge 3 servizi in parallelo

# Struttura:
# SNS Topic "ordini-topic"
#   |--- SQS "inventario-queue" --- Lambda inventario
#   |--- SQS "pagamenti-queue"  --- Lambda pagamenti
#   |--- SQS "notifiche-queue"  --- Lambda notifiche email

# Terraform
resource "aws_sns_topic" "ordini" {
  name = "ordini-topic"
}

resource "aws_sqs_queue" "inventario" {
  name = "inventario-queue"
  visibility_timeout_seconds = 60
  redrive_policy = jsonencode({
    deadLetterTargetArn = aws_sqs_queue.inventario_dlq.arn
    maxReceiveCount     = 3
  })
}

resource "aws_sns_topic_subscription" "ordini_inventario" {
  topic_arn = aws_sns_topic.ordini.arn
  protocol  = "sqs"
  endpoint  = aws_sqs_queue.inventario.arn

  # Filter policy: questa subscription riceve SOLO OrdineCreato
  filter_policy = jsonencode({
    eventType = ["OrdineCreato"]
  })
}
// SNS Publisher Java
import software.amazon.awssdk.services.sns.*;
import software.amazon.awssdk.services.sns.model.*;

public class SnsPublisher {

    private final SnsClient snsClient = SnsClient.builder()
        .region(Region.EU_WEST_1).build();

    public void publishOrdineCreato(String topicArn, OrdineCreato event) throws Exception {
        String messageJson = new ObjectMapper().writeValueAsString(event);

        PublishResponse response = snsClient.publish(
            PublishRequest.builder()
                .topicArn(topicArn)
                .message(messageJson)
                // Message attributes per subscription filter
                .messageAttributes(Map.of(
                    "eventType", MessageAttributeValue.builder()
                        .stringValue("OrdineCreato")
                        .dataType("String")
                        .build()
                ))
                // Subject visibile solo nelle email
                .subject("Nuovo ordine: " + event.getOrdineId())
                .build()
        );
        System.out.println("SNS Message ID: " + response.messageId());
    }
}

The Complete Comparison

Characteristic SQS SNS EventBridge
Model Point-to-point (queue) Fan-out (pub/sub) Content-based routing
Recipients 1 consumer at a time All registered in parallel Targets by rule (1-5)
Filter Nobody (takes everything) Message attributes (limited) Event pattern on any JSON field
Sorting Guaranteed FIFO (FIFO queue) No No
Deduplication Yes (FIFO queue) No No
Max. throughput Unlimited (Standard) Unlimited 10 million events/s per region
Latency Milliseconds (polling) < 1s (push) < 500ms
Schema Registry No No Yes (integrated)
Event Archive/Replay No No Si
Partner integration SaaS No No Yes (35+ partners)
Cost per million messages ~$0.40 ~$0.50 ~$1.00

Decision Guide: When to Use Which

Use SQS when:

  • You need to load balancing between multiple consumers of the same type (e.g. 5 Lambdas processing the same queue)
  • The message must be elaborated exactly once (FIFO queue)
  • You need to backpressure: Messages pile up in the queue when consumers are slow
  • Do you want the delivery delay (up to 15 minutes per single message)
  • You are integrating legacy systems that use the traditional queue model

Use SNS when:

  • The same event must reach multiple systems in parallel (fan-out pattern)
  • You need to send notifications (email, SMS, mobile push, HTTP webhook)
  • Combine SNS + SQS to have both fan-out and buffering for each consumer

Use EventBridge when:

  • You need to intelligent routing based on the contents of the payload
  • Integrate with AWS services (EC2, S3, RDS) or SaaS partners (Zendesk, Shopify, Stripe)
  • You want it Schema Registry and scheme governance
  • You need to Event Archive and Replay
  • Build a corporate event bus with centralized routing

SNS + SQS + Lambda Pattern: The Complete Stack

# Architettura tipica per un microservizio event-driven su AWS
#
# Client HTTP
#    |
#    v
# API Gateway / Lambda "ordini-api"
#    | pubblica su
#    v
# SNS Topic "ordini-events"
#    |-- filtra "OrdineCreato" --> SQS "inventario-queue" --> Lambda "inventario-handler"
#    |-- filtra "OrdineCreato" --> SQS "notifiche-queue"  --> Lambda "email-notifier"
#    |-- tutti i tipi         --> SQS "audit-queue"       --> Lambda "audit-logger"
#
# Ogni SQS ha la sua DLQ per messaggi non elaborabili
# Le Lambda usano le SQS come event source (SQS trigger)

# Configurazione Lambda con SQS trigger
resource "aws_lambda_event_source_mapping" "inventario" {
  event_source_arn = aws_sqs_queue.inventario.arn
  function_name    = aws_lambda_function.inventario_handler.arn
  batch_size       = 10         # processa 10 messaggi alla volta
  enabled          = true

  # Bisection on error: in caso di errore batch, divide il batch a meta
  # per identificare il messaggio problematico (evita il poison pill)
  bisect_batch_on_function_error = true

  # Finestra di aggregazione (utile per batch processing)
  maximum_batching_window_in_seconds = 5
}

EventBridge + SQS: the Best of Both Worlds

In many architectures, EventBridge and SQS are complementary: EventBridge does the intelligent routing, SQS does the buffering and load balancing. An event arrives on EventBridge, is routed to the appropriate SQS queue, and the consumer Lambda processes it from the queue with retry and DLQ. 

# EventBridge --> SQS (con transform di input opzionale)
resource "aws_cloudwatch_event_target" "ordini_to_sqs" {
  rule           = aws_cloudwatch_event_rule.ordini_vip.name
  event_bus_name = aws_cloudwatch_event_bus.mioapp.name
  arn            = aws_sqs_queue.vip_orders.arn

  # Input transformer: ristruttura il payload prima di inviarlo a SQS
  # Utile per estrarre solo i campi necessari
  input_transformer {
    input_paths = {
      ordineId  = "$.detail.ordineId"
      clienteId = "$.detail.clienteId"
      totale    = "$.detail.totale"
    }
    input_template = <<-EOF
    {
      "ordineId": "<ordineId>",
      "clienteId": "<clienteId>",
      "totale": "<totale>",
      "processato_da": "eventbridge-vip-rule"
    }
    EOF
  }
}

Next Steps in the Series

  • Article 8 – Dead Letter Queue and Resilience: the correct configuration of DLQ for SQS, SNS and EventBridge is critical to avoid message loss. Visibility timeout, maxReceiveCount and the reprocessing pattern.

Link with Other Series

  • AWS EventBridge (Article 6): in-depth analysis of event patterns, schema registry and event archive, all EventBridge configuration details.
  • Apache Kafka (Series 38): for on-premise or very high throughput use cases (>10 million msg/s), Kafka is superior to SQS/SNS. The choice depends on the cloud provider, by throughput and the need for event replay.