Charged Twice: How Idempotency Keys Stop Duplicate Payments

You tap "Pay". The spinner turns. And turns. Then the app says something went wrong. Did the money leave your account? You don't know. Neither, very often, does the app.

That moment happens at enormous scale. UPI processed 24.07 billion transactions in September 2026, an average of 802 million a day. If even 0.01% of those were retried after a timeout (that's my what-if, not a published figure), that would be 80,200 chances a day to charge someone twice.

The tool that prevents it is old, simple and still missing from a surprising amount of backend code: the idempotency key. This post covers why retries are dangerous, how the key works, the table and Spring Boot code to implement it, and the edge cases that bite.

A timeout tells you nothing

When a call to a payment service times out, one of three things happened:

1. The request never reached the server.

2. It reached the server, failed, and the error response was lost.

3. It reached the server, succeeded, and the success response was lost.

From the client's side, all three look identical. Retrying is correct in cases 1 and 2. In case 3, a blind retry charges the customer twice. And timeouts are not rare: mobile networks drop, load balancers time out, pods restart mid-request.

[IMG:retry-sequence]

What "idempotent" means here

An operation is idempotent if doing it twice has the same effect as doing it once. HTTP already defines GET, PUT and DELETE as idempotent. POST isn't, and "create a payment" is naturally a POST.

So we make the POST idempotent ourselves. The client attaches a unique key to the request. The server promises: for a given key, I will do the work at most once, and every repeat of that key gets the same answer.

The contract

There is an IETF draft that standardises this as the Idempotency-Key HTTP header, and payment APIs such as Stripe have used the same header for years. The rules are short:

• The client generates one key per user intent ("pay ₹1,499 for order 812"), not per HTTP attempt, and sends the same key on every retry of that intent. A UUID is fine.

• First time the server sees a key: process the request and store the response against the key.

• Same key again, request finished: return the stored response. Don't process anything.

• Same key, different request body: reject with 422. A key is bound to one request.

• Same key while the first request is still running: reject with 409 and let the client retry shortly.

• Keys expire. Stripe, for example, may remove keys once they are at least 24 hours old. After that, the same key is treated as a new request.

The table

CREATE TABLE idempotency_key (    client_id      VARCHAR(64)  NOT NULL,    idem_key       VARCHAR(100) NOT NULL,    request_hash   CHAR(64)     NOT NULL,    status         VARCHAR(20)  NOT NULL,      -- IN_PROGRESS or COMPLETED    response_code  INT,    response_body  JSONB,    locked_until   TIMESTAMPTZ  NOT NULL,    created_at     TIMESTAMPTZ  NOT NULL DEFAULT now(),    PRIMARY KEY (client_id, idem_key));

Three details matter. The primary key is scoped per client, so two clients can't collide on the same UUID by accident. request_hash is a SHA-256 of the canonical request body, so we can detect a key reused for a different payment. locked_until handles servers that crash mid-request, which we'll come to.

Claiming a key, atomically

The heart of the design is one statement. Whoever inserts the row first owns the key. The primary key makes the database the referee, so there's no window for two requests to both think they are first:

INSERT INTO idempotency_key (client_id, idem_key, request_hash, status, locked_until)VALUES (?, ?, ?, 'IN_PROGRESS', now() + interval '30 seconds')ON CONFLICT (client_id, idem_key) DO NOTHING;

If it inserted one row, this request owns the key and does the work. If it inserted zero rows, someone got there first, so read the existing row and decide.

The Spring Boot code

@PostMapping("/payments")public ResponseEntity<PaymentResponse> pay(@RequestHeader("Idempotency-Key") String key,                                           @AuthenticationPrincipal Client client,                                           @Valid @RequestBody PaymentRequest req) {    String hash = sha256(canonicalJson(req));    boolean claimed = jdbc.update("""            INSERT INTO idempotency_key (client_id, idem_key, request_hash, status, locked_until)            VALUES (?, ?, ?, 'IN_PROGRESS', now() + interval '30 seconds')            ON CONFLICT (client_id, idem_key) DO NOTHING            """, client.id(), key, hash) == 1;    if (!claimed) {        StoredKey existing = keyStore.find(client.id(), key);        if (!existing.requestHash().equals(hash)) {            return ResponseEntity.unprocessableEntity().build();          // 422        }        if (existing.inProgress()) {            return ResponseEntity.status(HttpStatus.CONFLICT).build();    // 409        }        return existing.toResponseEntity();                               // replay    }    PaymentResponse result = paymentGateway.charge(req, key);   // pass the key downstream    keyStore.complete(client.id(), key, 201, result);    return ResponseEntity.status(HttpStatus.CREATED).body(result);}

Notice what is not here: a @Transactional around the whole method. The claim has to commit immediately, so a concurrent retry sees IN_PROGRESS and gets a 409 instead of waiting on a lock. The claim and the completion are two separate, short writes.

Also notice paymentGateway.charge(req, key). The key travels downstream. That's what makes the next section work.

When the server crashes halfway

Suppose the charge succeeds and the pod dies before keyStore.complete runs. The row is stuck at IN_PROGRESS. The client retries and gets 409 forever. That's not acceptable.

This is what locked_until is for. Once it has passed, a retry may take the key over:

UPDATE idempotency_keySET locked_until = now() + interval '30 seconds'WHERE client_id = ? AND idem_key = ?  AND status = 'IN_PROGRESS' AND locked_until < now();

One row updated means this request now owns the key and runs the charge again. That's only safe because the same key went to the payment gateway. A gateway that supports idempotency recognises the key and returns the original charge instead of creating a second one. If a downstream system doesn't support keys, you need a different recovery: look the payment up by your own reference before retrying.

[IMG:key-states]

The consumer side: Kafka

Since Kafka 3.0, producers default to idempotence, which stops a producer retry from writing the same message twice. That's a good default, but it says nothing about your consumer. Kafka delivers at least once, so a consumer can and will see the same event twice after a rebalance or a crash.

Kafka's exactly-once (transactions plus read_committed) covers reading from Kafka and writing back to Kafka. The moment your consumer writes to Postgres, you need de-duplication on your side:

CREATE TABLE processed_event (    consumer  VARCHAR(100) NOT NULL,    event_id  UUID         NOT NULL,    PRIMARY KEY (consumer, event_id));

@KafkaListener(topics = "payment-captured")@Transactionalpublic void onPaymentCaptured(PaymentCaptured event) {    int inserted = jdbc.update("""            INSERT INTO processed_event (consumer, event_id) VALUES ('ledger', ?)            ON CONFLICT DO NOTHING            """, event.eventId());    if (inserted == 0) {        return;                       // seen it before    }    ledger.credit(event.merchantId(), event.amount());}

The marker row and the business write are in the same database transaction, so they commit or roll back together. A crash between them can't leave you with one and not the other.

The reverse problem, writing to the database and publishing an event, has the same shape. Write the event to an outbox table in the same transaction as the business change, and let a separate relay publish it. Otherwise you get "database committed, publish failed", and the rest of the system never hears about the payment.

Every case, in one table

Test it the way production will break it

Fire the same request, with the same key, from twenty threads at once, and assert there is exactly one charge:

@Testvoid sameKeyFromTwentyThreadsChargesOnce() throws Exception {    String key = UUID.randomUUID().toString();    CountDownLatch go = new CountDownLatch(1);    try (var pool = Executors.newVirtualThreadPerTaskExecutor()) {        for (int i = 0; i < 20; i++) {            pool.submit(() -> { go.await(); return client.pay(key, samplePayment()); });        }        go.countDown();    }    verify(paymentGateway, times(1)).charge(any(), eq(key));}

Expect some of those calls to return 409. That's the design working.

The checklist

• Every endpoint that moves money or creates something accepts an Idempotency-Key.

• The key is claimed with a single INSERT behind a unique constraint.

• Replays return the stored response; mismatched bodies get 422; in-flight duplicates get 409.

• Stuck keys can be reclaimed after a lease, and the key is passed downstream.

• Every Kafka consumer that writes to a database de-duplicates by event ID in the same transaction.

• Events that must follow a database change go through an outbox.

Payment webhooks need exactly the same treatment. I covered that, along with reservations that expire before the payment lands, in my flash sale post.

Where does your system create the idempotency key: the client, the gateway or the server? I'm curious what's worked for you.