Message Queue

What is a message queue?

A message queue (SQS, RabbitMQ, or the log-structured Kafka family) lets services communicate without being simultaneously available: producers append messages and move on; consumers pull and process at their own rate. That temporal decoupling changes failure behavior fundamentally — a payment-processor slowdown becomes a growing queue and a delay, rather than cascading timeouts through every upstream caller. Spikes become backlogs; backlogs drain.

The contract you must design for

Practical queues deliver at-least-once: after a crash between processing and acknowledgment, the message returns. Duplicates are therefore not an edge case but a specification — every consumer must be idempotent (dedupe keys, upserts, “already processed” checks), or you will double-charge someone during your first bad deploy. The second obligatory pattern is the dead-letter queue: messages failing N attempts route aside for inspection instead of poisoning the queue in an infinite retry loop. A queue without a DLQ is an outage scheduled for later.

Queue or log? The fork that matters

Classic queues (SQS/RabbitMQ) delete on consumption — one message, one processing, work-distribution semantics. Log systems (Kafka) retain messages; consumers track positions, multiple independent consumers replay the same stream. Choose queues for task distribution, logs for event streams that several systems consume or that need replay. Teams that pick Kafka “for scale” and use it as a task queue inherit its operational weight for none of its benefits.

What people get wrong

• Assuming ordering — standard queues don’t guarantee it; FIFO variants do, at throughput cost per ordering key.
• No backpressure story: if producers permanently outpace consumers, the queue is a slow-motion failure; autoscale consumers on queue depth and alert on age-of-oldest-message.
• Payloads as blobs — unversioned message schemas make every producer deploy a potential consumer-crash event.

Primary source: AWS SQS documentation — best practices

Queues vs. Pub/Sub Topics

Message queues are often discussed alongside publish/subscribe (pub/sub) systems, but the two serve different delivery patterns: a classic queue delivers each message to exactly one consumer (useful for distributing work across a pool of workers, where each task should be handled once), while a pub/sub topic delivers each published message to every subscriber independently (useful for broadcasting an event to multiple interested systems that each need to react to it separately). Many modern messaging platforms, including Amazon SNS combined with SQS, support both patterns, and choosing the right one for a given integration is a foundational design decision — using a pub/sub pattern where a queue’s single-consumer semantics were actually needed (or vice versa) is a common source of subtle bugs in distributed system design.