Exactly-once effects on an at-least-once queue

I published a small reference implementation this week: idempotent-consumer, a Java 21 / Spring Boot service that shows the pattern I actually run in production to stop duplicate SQS deliveries from becoming duplicate debits — no FIFO queues, no distributed locks.

The problem: FinEdge Bank, pre-debit charge recovery

FinEdge recovers service charges — SMS alert fees, cheque-book fees, locker rent — from customer accounts. An upstream billing system publishes a ChargeDueEvent to SQS for every charge that’s due. A consumer picks each event up and, for that charge:

1. debits the customer’s account via the core-banking API, and
2. sends a pre-debit notification.

SQS is at-least-once by design: a visibility-timeout expiry, a consumer crash mid-processing, or a DLQ redrive will all redeliver a message that was already handled. At FinEdge’s volume — roughly 2M charges/day, bursty around month-end when most fees post — redeliveries aren’t an edge case, they’re a daily occurrence.

A duplicate delivery that causes a duplicate debit is not a bug ticket. It’s a customer charged twice for a ₹25 SMS fee, a regulatory complaint, and a manual reversal process. The fix has to be structural, not “try to be careful in the handler.”

Two constraints on the fix, from how FinEdge runs its queues:

• No FIFO queues. FIFO’s per-message-group throughput ceiling doesn’t work at this volume, especially at month-end peaks.
• No distributed locks. A lock service is another thing to run, monitor, and page on. Whatever protects against duplicates has to be a property of storage already in the stack, not a new moving part.

How it works

Before doing anything, the consumer makes a claim: a DynamoDB PutItem on the charge’s business key, guarded by a ConditionExpression that only succeeds if the key is unclaimed or its claim has expired. That claim is atomic — DynamoDB evaluates the condition server-side, so under concurrent delivery exactly one caller can win it. Only the winner runs the debit and the notification; everyone else gets told what to do with their copy of the message instead of guessing.

flowchart TD
    A[SQS: ChargeDueEvent] --> B["claim: PutItem chargeId\nConditionExpression:\nattribute_not_exists(pk)\nOR (state=IN_PROGRESS AND leaseUntil < now)"]
    B --> C{claim result}
    C -- "ACQUIRED" --> D[debit via core-banking API]
    D -- "success" --> E[send pre-debit notification]
    D -- "permanent failure" --> F["markFailed\n(ConditionExpression: state=IN_PROGRESS)"]
    E --> G["markDone\n(ConditionExpression: state=IN_PROGRESS)"]
    G --> H[ack: delete from queue]
    F --> I[forward to DLQ, then ack]
    C -- "DUPLICATE_DONE\n(already completed)" --> J[log for audit, ack: delete]
    C -- "DUPLICATE_IN_PROGRESS\n(lease still live)" --> K[do NOT ack]
    K --> L[visibility timeout expires]
    L --> M[SQS redelivers]
    M --> B
    C -- "DUPLICATE_FAILED\n(previously failed)" --> I

And the sequence for the case this whole repo exists to handle — the same charge delivered twice while the first delivery is still in flight:

sequenceDiagram
    participant SQS
    participant W1 as Worker A (1st delivery)
    participant DDB as DynamoDB (idempotency table)
    participant CB as Core Banking
    participant W2 as Worker B (redelivery)

    SQS->>W1: ChargeDueEvent(chargeId=X)
    W1->>DDB: PutItem X, state=IN_PROGRESS (conditional)
    DDB-->>W1: ACQUIRED
    W1->>CB: debit(X, ...)
    Note over SQS,W2: visibility timeout expires before W1 finishes -> redelivered
    SQS->>W2: ChargeDueEvent(chargeId=X)
    W2->>DDB: PutItem X, state=IN_PROGRESS (conditional)
    DDB-->>W2: ConditionalCheckFailedException (state=IN_PROGRESS, lease live)
    W2-->>SQS: do NOT ack
    CB-->>W1: debit succeeded
    W1->>DDB: UpdateItem X, state=DONE (conditional: state=IN_PROGRESS)
    DDB-->>W1: OK
    Note over SQS,W2: SQS redelivers X again later
    SQS->>W2: ChargeDueEvent(chargeId=X)
    W2->>DDB: PutItem X (conditional)
    DDB-->>W2: ConditionalCheckFailedException (state=DONE)
    W2-->>SQS: ack (drop, audited) — no second debit

Why a conditional write, not a cache/get-then-act check

The naive fix — “check a cache or table for the key, and only proceed if it’s not there” — has a race built in: two workers can both read “not seen” before either writes anything, and both proceed. Closing that race with an external lock (Redis SETNX with lock-service semantics, ZooKeeper, etc.) just relocates the coordination problem to a system you now also have to run and keep available.

A DynamoDB conditional PutItem doesn’t have that race, because there is no separate read step to race against — the check and the write are the same atomic server-side operation.

The state machine

The idempotency table has one item per chargeId, and the item’s state plus a lease (an expiry timestamp on the claim) is the entire coordination mechanism. The lease exists so a worker that crashes after claiming but before finishing doesn’t poison the key forever — a later redelivery can reclaim an expired lease, atomically, the same way it claims a fresh key.

What “exactly-once” means here — honestly

This gives exactly-once recorded effect, not exactly-once delivery — nothing built on top of SQS gives you that. There is a real crash window: if the process dies after the debit succeeds but before markDone commits, the key is still IN_PROGRESS. Once its lease expires, a redelivery will reclaim it and run the debit again.

The mitigation lives at the boundary, not in this service: AccountDebitPort implementations must pass chargeId through as the core-banking API’s own idempotency key. Most core-banking debit APIs accept one for exactly this reason. When that’s true, the crash-window retry reaches the core-banking system a second time, and it is the one that recognizes the key and no-ops — this service’s job is to make that retry rare and bounded (bounded by the lease duration), not to be the last line of defense on its own.

One more honest edge, by design rather than oversight: a debit that succeeds but whose notification subsequently fails does not roll back or retry the debit. Money already moved; that’s the fact that matters, and it’s what’s recorded (ChargeStatus.DEBITED_NOTIFICATION_FAILED). Retrying the whole charge on a notification failure would risk a second debit to save a notification — the wrong trade every time in a banking context. A missed notification is an operational follow-up; a duplicate debit is an incident.

Finally: the idempotency key is the business chargeId from the billing system, never the SQS messageId. The messageId is a transport artifact — it’s different on every redelivery and every producer retry, so keying on it would defeat deduplication entirely. chargeId identifies the business fact (“this charge is owed”), which is the granularity “don’t do this twice” actually needs.

Why Spring Cloud AWS for the SQS side

The DynamoDB adapter talks to the SDK directly — full control over ConditionExpression matters there, and there’s no listener lifecycle to manage. For SQS, the repo uses Spring Cloud AWS’s @SqsListener rather than hand-rolling a polling loop on the raw SDK client:

• Manual-ack mode (acknowledgementMode = MANUAL) maps directly onto the four-way outcome above — ack, don’t-ack, and forward-then-ack are all explicit, one-line decisions in the listener, with no implicit auto-ack to fight.
• Concurrency, polling, and shutdown are handled by the container; the adapter code is exactly what the hexagonal boundary says it should be — deserialize, delegate, map the outcome to an ack decision.
• It’s the standard, actively maintained way to consume SQS in a Spring Boot app; a hand-rolled poller would be re-implementing the same lifecycle with more code and no more control where it matters (the DynamoDB write).

Layout

domain/       framework-free: model (Charge, Money, ...), ports (interfaces), the claim→debit→notify→done service
application/  DTO<->domain mapping and the use-case entry point; owns the wire-format DTO
adapter/in/   SQS listener — deserialize, delegate, map outcome to ack/no-ack/DLQ. No business logic.
adapter/out/  DynamoDB idempotency store; fake core-banking debit + notification adapters
config/       Spring wiring, LocalStack-aware AWS client setup

The domain package has zero Spring or AWS imports — every dependency it needs is a port it defines and an adapter implements, so ChargeRecoveryService is testable with plain fakes and has no idea DynamoDB or SQS exist.

Try it

Requires Docker, Java 21, Maven, and (optionally) awslocal.

make demo

demo is the whole proof in one command: it stops any app run already in progress, starts LocalStack, resets the idempotency table and both queues to a clean slate, builds and runs the app, sends 20 charges with ~30% resent under the same chargeId to simulate redelivery, and verifies every chargeId was debited exactly once. It’s hermetic — make demo && make demo passes back-to-back — because it resets state before it ever sends a message, rather than assuming the table is already empty.

Tests

• Unit (ChargeRecoveryServiceTest, fakes only, no Docker): duplicate-drop, in-flight-not-acked, expired-lease reclaim, permanent-failure → FAILED, and the notification-failure-doesn’t-touch-the-debit case.
• Concurrency: 50 threads hit one chargeId simultaneously; exactly one debit runs.
• Integration (ChargeRecoveryIT, Testcontainers LocalStack, needs Docker): the same message sent five times through real SQS into the real listener and a real DynamoDB table; asserts one debit and one DONE record.

MIT licensed. Questions or war stories: LinkedIn.