RabbitMQ consumers are easy to demonstrate and deceptively difficult to operate correctly.
A basic example typically creates a connection, declares a queue, subscribes to messages, and acknowledges each delivery after handling it. That code can be perfectly valid while still causing duplicate payments, repeated emails, inconsistent projections, inventory corruption, or hard-to-debug production incidents.
The underlying reason is simple:
In a distributed system, successful processing and successful message acknowledgement are not one atomic operation.
RabbitMQ commonly provides at-least-once delivery semantics. That is usually the correct trade-off: a message is not silently lost when a consumer crashes, loses its connection, or fails before confirming the delivery. The consequence is that consumers must be prepared to receive a logically identical message more than once.
This article explains how to design .NET RabbitMQ consumers that remain correct under duplicate delivery, retries, consumer crashes, redeliveries, transient infrastructure failures, poison messages, concurrent consumption, and external side effects.
The goal is not to claim “exactly once” processing. In most real systems, that claim is either incomplete or false.
Make duplicate delivery harmless, visible, and operationally manageable.
The failure window that creates duplicates
Consider a consumer handling an OrderPaid event:
- RabbitMQ delivers
OrderPaid. - The consumer writes a row to the database.
- The consumer sends a manual acknowledgement to RabbitMQ.
- RabbitMQ removes the message from the queue.
Now consider a failure between steps 2 and 3.
RabbitMQ -> Consumer -> Database commit -> Consumer crashes
\
Acknowledgement never reaches RabbitMQ
When the connection closes, RabbitMQ requeues the unacknowledged message. Another consumer—or the restarted process—receives the same event again.
The database write has already happened. The message is delivered again anyway.
The reverse ordering is worse:
- The consumer acknowledges the message.
- The consumer writes to the database.
- The process crashes before the database transaction commits.
The message is now gone, but the business operation never happened.
Acknowledgement first: RabbitMQ -> Consumer -> ACK succeeds -> Process crashes -> Database write lost
For durable business workflows, acknowledgement must occur after successful processing. This protects against message loss, but it necessarily permits duplicate delivery.
That is not a RabbitMQ flaw. It is a fundamental distributed-systems trade-off.
“Exactly once” is usually an incomplete promise
A message broker can track whether it redelivered a message. A database can enforce a unique constraint. A payment provider can accept an idempotency key.
None of those guarantees exactly-once behavior across every system boundary by themselves.
For example, imagine this workflow:
Message received -> Database transaction committed -> HTTP request sent to payment provider -> Consumer crashes before recording success
On retry, the consumer cannot know whether the remote provider received and completed the request unless the external operation itself supports idempotency.
A more precise vocabulary is useful:
| Term | Meaning |
| At-most-once | A message may be lost, but is not intentionally retried. |
| At-least-once | A message is retried until acknowledged; duplicates are possible. |
| Effectively-once | Duplicates may arrive, but the observable business effect is applied once. |
| Exactly-once | Every effect occurs once across all participating systems. Rare, expensive, and frequently overstated. |
For most application consumers, effectively-once business behavior is the achievable and useful objective.
Define idempotency in business terms
A consumer is idempotent when processing the same logical message repeatedly produces the same valid business outcome as processing it once.
This definition is intentionally broader than “avoid duplicate inserts.”
Naturally idempotent operations
Some operations can be expressed as state assignment:
customer.IsVerified = true;Processing the same verification event ten times still leaves IsVerified as true.
Likewise:
UPDATE orders
SET status = 'Paid'
WHERE id = @OrderId
AND status <> 'Paid';This can be safe when the state transition is valid and no other side effect is triggered.
Non-idempotent operations
Other operations accumulate effects:
account.CreditBalance += message.Amount;or:
INSERT INTO loyalty_points (customer_id, points)
VALUES (@CustomerId, @Points);Those operations are unsafe under duplicate delivery unless they are guarded by a durable deduplication mechanism.
Business identity versus broker delivery identity
RabbitMQ delivery tags are scoped to a channel. They are not stable business identifiers and must not be used as a durable idempotency key.
A message needs a stable, producer-generated identity:
{
"messageId": "018f8b12-0e9b-79e6-a697-7234da683bd6",
"eventType": "billing.invoice-paid.v1",
"occurredAtUtc": "2026-09-10T13:00:00Z",
"payload": {
"invoiceId": "inv_10394",
"customerId": "cus_832",
"amount": 149.00,
"currency": "USD"
}
}A robust message envelope should normally include messageId, eventType, occurredAtUtc, correlationId, causationId, schemaVersion, and the business payload.
If producers cannot reliably provide a unique messageId, derive an idempotency key from immutable business facts—for example invoiceId + "paid"—but only where that truly represents one business event.
The transactional inbox pattern
The most broadly useful solution is a transactional inbox.
The consumer stores the message identity in the same database transaction as the business changes it performs. A uniqueness constraint guarantees that only one transaction can claim a given message for that consumer.
RabbitMQ delivery
|
v
Database transaction
1. Insert message identity into inbox
2. Apply business state changes
3. Insert outgoing events into outbox, if necessary
|
v
Commit transaction
|
v
Acknowledge RabbitMQ delivery
If processing fails before commit, both the inbox entry and business changes roll back. RabbitMQ can safely redeliver the message.
If the process crashes after commit but before acknowledgement, RabbitMQ redelivers it. The next consumer sees the inbox entry and treats the message as already handled.
Inbox schema
A simple PostgreSQL inbox table can look like this:
CREATE TABLE consumer_inbox (
consumer_name text NOT NULL,
message_id uuid NOT NULL,
event_type text NOT NULL,
correlation_id text NULL,
received_at_utc timestamptz NOT NULL DEFAULT now(),
processed_at_utc timestamptz NOT NULL DEFAULT now(),
CONSTRAINT pk_consumer_inbox
PRIMARY KEY (consumer_name, message_id)
);The composite key matters. A message can legitimately be processed by different consumers. BillingProjectionConsumer and CustomerNotificationConsumer should each have their own deduplication boundary.
The correct identity is therefore usually consumer name + message ID, not only message ID.
Retention policy
An inbox table grows continuously. Deleting data too aggressively reintroduces duplicate-processing risk; retaining everything forever increases storage and index costs.
Retention should be derived from actual operational behavior:
maximum broker retention + maximum retry delay + disaster-recovery replay window + safety margin
For many systems, 30 to 90 days is reasonable. Financial, audit-heavy, or event-replay systems may need longer retention.
Do not schedule inbox cleanup before understanding replay procedures. A six-month-old event replayed after a migration can become dangerous if the deduplication history was deleted after seven days.
A production-oriented .NET implementation
The following example shows the key transactional structure. It uses PostgreSQL as the durable idempotency store, but the principle is identical for SQL Server, MySQL, or another transactional database.
using System.Text.Json;
using Npgsql;
using RabbitMQ.Client;
using RabbitMQ.Client.Events;
public sealed class InvoicePaidConsumer
{
private const string ConsumerName = "billing.invoice-paid.v1";
private readonly NpgsqlDataSource _dataSource;
private readonly ILogger<InvoicePaidConsumer> _logger;
public InvoicePaidConsumer(
NpgsqlDataSource dataSource,
ILogger<InvoicePaidConsumer> logger)
{
_dataSource = dataSource;
_logger = logger;
}
public async Task HandleAsync(
IChannel channel,
BasicDeliverEventArgs delivery,
CancellationToken cancellationToken)
{
InvoicePaidEvent message;
try
{
message = JsonSerializer.Deserialize<InvoicePaidEvent>(
delivery.Body.Span,
SerializerOptions)
?? throw new InvalidOperationException(
"InvoicePaid message body was empty.");
}
catch (JsonException exception)
{
_logger.LogError(
exception,
"Invalid JSON received on invoice-paid queue. Delivery tag: {DeliveryTag}",
delivery.DeliveryTag);
await RejectToDeadLetterQueueAsync(
channel,
delivery.DeliveryTag,
cancellationToken);
return;
}
try
{
var result = await ProcessTransactionallyAsync(
message,
cancellationToken);
if (result == ProcessingResult.AlreadyProcessed)
{
_logger.LogInformation(
"Ignoring duplicate message {MessageId} for consumer {ConsumerName}.",
message.MessageId,
ConsumerName);
}
await channel.BasicAckAsync(
delivery.DeliveryTag,
multiple: false,
cancellationToken);
}
catch (TransientProcessingException exception)
{
_logger.LogWarning(
exception,
"Transient failure while handling message {MessageId}. It will be retried.",
message.MessageId);
await RequeueAsync(
channel,
delivery.DeliveryTag,
cancellationToken);
}
catch (Exception exception)
{
_logger.LogError(
exception,
"Permanent processing failure for message {MessageId}.",
message.MessageId);
await RejectToDeadLetterQueueAsync(
channel,
delivery.DeliveryTag,
cancellationToken);
}
}
private async Task<ProcessingResult> ProcessTransactionallyAsync(
InvoicePaidEvent message,
CancellationToken cancellationToken)
{
await using var connection =
await _dataSource.OpenConnectionAsync(cancellationToken);
await using var transaction =
await connection.BeginTransactionAsync(cancellationToken);
var inserted = await TryInsertInboxRecordAsync(
connection,
transaction,
message,
cancellationToken);
if (!inserted)
{
await transaction.RollbackAsync(cancellationToken);
return ProcessingResult.AlreadyProcessed;
}
await MarkInvoiceAsPaidAsync(
connection,
transaction,
message,
cancellationToken);
await InsertOutboxEventAsync(
connection,
transaction,
new InvoicePaymentRecordedEvent(
MessageId: Guid.NewGuid(),
InvoiceId: message.InvoiceId,
PaidAtUtc: message.PaidAtUtc),
cancellationToken);
await transaction.CommitAsync(cancellationToken);
return ProcessingResult.Processed;
}
}The essential property is not the exact library API. It is the transaction boundary:
inbox insert + domain mutation + outbox insert = one database transaction
The RabbitMQ acknowledgement occurs only after that transaction commits.
Why the uniqueness constraint is non-negotiable
A common but unsafe implementation first queries the inbox:
var exists = await inboxRepository.ExistsAsync(message.MessageId);
if (!exists)
{
await ProcessAsync(message);
await inboxRepository.InsertAsync(message.MessageId);
}Two consumers can execute this concurrently:
Consumer A: checks inbox -> not found Consumer B: checks inbox -> not found Consumer A: processes message Consumer B: processes message
The bug is a classic time-of-check-to-time-of-use race.
The database must decide who owns the message. A unique index or primary key does exactly that. The insert attempt becomes the atomic claim operation.
Manual acknowledgements are mandatory for durable workflows
Auto-acknowledgement tells RabbitMQ that a message has been successfully handled as soon as it is delivered to the consumer.
That may be appropriate for telemetry, best-effort analytics, or disposable cache invalidation signals. It is not appropriate for workflows where data loss matters.
- Disable automatic acknowledgements.
- Acknowledge only after durable success.
- Reject invalid or permanently unprocessable messages.
- Route rejected messages to a dead-letter exchange.
- Make redeliveries observable.
An acknowledgement is not a “received” signal. It is a durable statement that the consumer has completed all work it is responsible for doing for this message.
Retry topology: do not immediately requeue forever
A transient database timeout should be retried. A malformed message should not. A message that violates a business invariant may require human intervention.
Blindly calling BasicNack(requeue: true) on every error creates hot loops:
Consumer fails -> message requeued immediately -> same consumer receives it again -> fails again -> CPU usage rises, logs explode, useful work stalls
A better topology separates retryable delivery failures from terminal failures.
orders.events
|
v
orders.invoice-paid.queue
|
+-- transient failure --> orders.invoice-paid.retry.1m
| |
| +-- TTL expiry --> original queue
|
+-- retry exhausted --> orders.invoice-paid.dead-letter
A typical retry setup uses a primary queue, one or more retry queues with TTL, a dead-letter exchange, and a parking-lot queue for messages requiring investigation.
| Failure | Recommended action |
| Database timeout or temporary network issue | Retry with delay |
| Dependency rate limit | Retry with backoff, respecting Retry-After if available |
| Invalid JSON | Dead-letter immediately |
| Unknown schema version | Dead-letter and alert |
| Validation failure caused by producer defect | Dead-letter and create an operational incident |
| External provider duplicate response | Treat as success if the idempotency contract confirms prior completion |
The error taxonomy must be deliberate. Treating every exception as transient hides producer defects and creates noisy retry storms.
The outbox pattern protects downstream publication
The transactional inbox protects a consumer from duplicate input. It does not solve the next problem:
Database transaction commits -> Consumer publishes next event to RabbitMQ -> Process crashes before publish
The business state now changed, but downstream services never receive the event. Publishing first is not safe either, because downstream consumers may observe an event for a state change that did not persist.
The transactional outbox pattern resolves this by writing the outbound event to an outbox_messages table in the same transaction as the business state:
Transaction: - Record inbox message - Update domain state - Insert outbound event in outbox Commit Outbox publisher: - Reads unpublished outbox rows - Publishes with publisher confirms - Marks rows as published
The outbox publisher may itself publish a message more than once if it crashes after RabbitMQ accepts the publication but before the database records success. That is expected. Downstream consumers must still be idempotent.
Every consumer is idempotent, even when every producer uses an outbox.
External side effects need their own idempotency contract
Database transactions cannot atomically commit an HTTP request, a payment-provider call, an email send, or a file upload.
For external effects, use a provider-supported idempotency key whenever possible.
POST /payments
Idempotency-Key: 018f8b12-0e9b-79e6-a697-7234da683bd6The key should be stable for the logical operation, not regenerated per retry.
Avoid embedding an unprotected external call in the middle of a consumer transaction. It increases transaction duration, holds database resources while waiting on networks, and leaves uncertain outcomes after timeouts.
Consumer concurrency and prefetch are correctness concerns
prefetch is frequently treated as a throughput-only tuning parameter. It also influences failure blast radius, fairness, memory pressure, and shutdown behavior.
Prefetch = 500 Consumer receives 500 messages Consumer process crashes RabbitMQ requeues 500 messages
That may be acceptable for fast, idempotent handlers. It is risky for expensive consumers with slow database transactions or rate-limited external dependencies.
Start with a modest prefetch value and measure:
prefetch ≈ consumer concurrency × expected work in flight
If a process has eight workers and each message requires a database transaction plus a small amount of CPU work, a prefetch between 16 and 64 is a reasonable point to test—not a universal answer.
RabbitMQ queues are ordered, but concurrent consumers and retries alter observed processing order. If events for the same aggregate must be processed sequentially, partition by aggregate key, enforce optimistic concurrency, version events per aggregate, or design handlers to be commutative where possible.
Graceful shutdown prevents unnecessary redelivery
A consumer should stop receiving new work before closing its connection, then allow in-flight work to finish within a bounded timeout.
- Mark the consumer instance as draining.
- Cancel consumption or stop accepting new deliveries.
- Wait for in-flight handlers to finish.
- Acknowledge successfully completed deliveries.
- Close the channel and connection.
- Allow uncompleted deliveries to be redelivered.
For ASP.NET Core hosted services, connect this flow to the application cancellation token. Kubernetes termination grace periods must be longer than the worst-case expected in-flight processing time, including database retry behavior.
Observability: duplicates should be measurable, not surprising
A reliable consumer is not only correct. It is diagnosable during an incident.
At minimum, log these fields consistently:
message_id event_type consumer_name correlation_id causation_id rabbitmq_redelivered retry_attempt queue_name processing_duration_ms processing_result
| Metric | Why it matters |
consumer_messages_processed_total | Throughput and success/failure totals |
consumer_duplicates_total | Detects redelivery patterns and publisher issues |
consumer_processing_duration_seconds | Reveals latency changes and capacity constraints |
consumer_retry_total | Detects transient dependency instability |
consumer_dead_letter_total | Requires operational attention |
rabbitmq_queue_messages_ready | Measures backlog |
rabbitmq_queue_messages_unacknowledged | Measures in-flight work |
rabbitmq_queue_oldest_message_age_seconds | More useful than count for freshness-sensitive systems |
outbox_pending_messages | Detects downstream publishing problems |
outbox_oldest_unpublished_age_seconds | Alerts before integration delays become incidents |
Useful alerts include dead-letter messages greater than zero for critical workflows, oldest message age exceeding the service-level objective, retry rate increasing beyond baseline, duplicate rate spikes, outbox age growth, no successful consumption for a critical queue, and queue depth growing while consumer replicas are healthy.
Message contracts must evolve safely
Consumers should not assume event contracts remain static forever.
Prefer versioned event types:
billing.invoice-paid.v1 billing.invoice-paid.v2
Additive changes are usually safer. Removing or changing the meaning of existing fields is much more dangerous.
- Publish a new event version.
- Support both versions during migration.
- Move producers and consumers deliberately.
- Monitor old-version traffic.
- Retire the previous version only after consumers no longer need it.
An event is a contract between independently deployable systems. Treat it with the same discipline as a public API.
A production readiness checklist
Delivery and idempotency
- The producer supplies a stable
messageId. - The consumer uses manual acknowledgements.
- Acknowledgement happens only after durable processing succeeds.
- Duplicate delivery is safe by design.
- The database enforces deduplication with a unique constraint.
- Inbox records and business changes are committed atomically.
- Inbox data retention covers replay and recovery windows.
Failure handling
- Transient and permanent errors are classified separately.
- Retries use delayed backoff rather than immediate hot-loop requeues.
- Invalid payloads are dead-lettered.
- Dead-letter queues are monitored and have an investigation process.
- Poison-message handling does not silently discard business data.
Downstream consistency
- Outbound events use a transactional outbox.
- Outbox publishing uses RabbitMQ publisher confirms.
- External providers receive stable idempotency keys where supported.
- Every downstream consumer is independently idempotent.
Operations
- Logs include message and correlation identifiers.
- Metrics cover throughput, duplicates, retries, dead letters, lag, and outbox age.
- Alerts are based on age, failure trends, and business criticality.
- Shutdown drains in-flight work safely.
- Prefetch and concurrency were measured under realistic load.
- Event schema evolution has a compatibility strategy.
Final thoughts
At-least-once delivery is not a problem to eliminate. It is a property to design around.
The dangerous consumer is not the one that receives duplicates; duplicates are inevitable under realistic failures. The dangerous consumer is the one that treats duplicate delivery as impossible.
A production-grade .NET RabbitMQ consumer should assume every message can arrive more than once, durably claim each message through an inbox uniqueness constraint, process business state within the same transaction, acknowledge only after commit, publish downstream work through an outbox, retry transient failures with controlled backoff, dead-letter permanent failures visibly, and make every stage observable.
When these principles are implemented consistently, consumer crashes, retries, deployment interruptions, connection failures, and broker redeliveries stop being exceptional edge cases. They become normal, recoverable parts of the system’s operating model.
Bir yanıt yazın