Idempotency Keys Explained: How to Make API Retries Safe

A client sends a request to create a payment. The server charges the card, writes the row, and starts sending back a 201. Somewhere between the server and the client, the connection drops. From the client's point of view, nothing happened: it got a timeout, not a response. The obvious move is to retry. If the API has no way to recognize that second request as a repeat of the first, the customer is now charged twice.

Idempotency keys are the standard fix for this. The client attaches a unique identifier to an operation, the server remembers what it did the first time it saw that identifier, and every later request carrying the same key gets the original result instead of a fresh side effect. The idea is simple. The details, which decide whether it actually protects you, are where most implementations go wrong.

Why HTTP methods alone do not solve it

RFC 9110 defines a method as idempotent when the intended effect of several identical requests is the same as the effect of one. PUT, DELETE and the safe methods such as GET qualify. POST does not. The specification draws the practical consequence directly: a client should not automatically retry a non-idempotent request unless it has some way to know the operation is actually safe to repeat, or that the original was never applied. It also says a proxy must not automatically retry non-idempotent requests.

That leaves a gap. Creating things is usually a POST, and creating things is exactly where duplicates hurt: orders, payments, emails, provisioned servers. An idempotency key is the "some way to know" that RFC 9110 asks for. It turns a POST into an operation the server can safely deduplicate.

How the major APIs define it

Stripe's API is the reference most teams copy. Clients send an Idempotency-Key header on POST requests. Stripe suggests V4 UUIDs or another random string with enough entropy, allows keys up to 255 characters, and advises against putting sensitive data such as email addresses in them. For its v1 API, Stripe stores the status code and body of the first request once endpoint execution begins, and replays that saved response on retries, including 500 errors. Keys can be pruned once they are at least 24 hours old, after which a reused key is treated as a new request. Incoming parameters are compared with the original, and a mismatch is an error.

AWS uses the same pattern under a different name. Its Builders' Library article on idempotent APIs describes a caller-provided client request token, such as the ClientToken parameter on EC2's RunInstances. The AWS SDKs and CLI generate one when the caller omits it and reuse it across their own retries. A retry returns a semantically equivalent response, which may show newer state, such as an instance that has moved from pending to running, while still describing the same resource.

The IETF HTTPAPI working group tried to standardize the header. Its draft, draft-ietf-httpapi-idempotency-key-header, reached version 07 in October 2025 and is now listed as expired rather than published as an RFC, so treat it as a well-reasoned design reference rather than a standard. It specifies the Idempotency-Key header as a Structured Field string, targets POST and PATCH, and recommends three error responses:

• 400 Bad Request when an endpoint that requires a key receives a request without one.
• 422 Unprocessable Content when a key is reused with a different request payload.
• 409 Conflict when a retry arrives while the original request is still being processed.

The four states a key can be in

Every incoming request with a key falls into one of four cases, and a correct server handles each one explicitly:

1. New key: claim it atomically, do the work, and store the response against the key.
2. Known key, same request, finished: return the stored status and body without doing any work.
3. Known key, same request, still running: return 409 so the client backs off and tries again later, instead of starting a second copy of the work.
4. Known key, different request: return 422. The client has a bug, and silently replaying an unrelated response would hide it.

Detecting the fourth case needs a fingerprint of the request. The IETF draft suggests a checksum of the payload or selected fields. A common approach is a SHA-256 hash of the body serialized as canonical JSON, with sorted keys and fixed separators, so that two semantically identical bodies produce the same hash regardless of field order.

A worked example in Python and SQLite

The program below implements the four cases with nothing but the Python standard library. It uses an in-memory SQLite database so it runs as is. The key detail is that claiming the key is a single INSERT with ON CONFLICT DO NOTHING, so the database, not application code, decides which request wins a race. The payment insert and the stored response are then committed in one transaction, so there is never a payment without a recorded response or a response without a payment.

import hashlib, json, sqlite3, time, uuid

db = sqlite3.connect(":memory:", isolation_level=None)
db.executescript("""
CREATE TABLE idempotency_keys (
  key           TEXT PRIMARY KEY,
  fingerprint   TEXT NOT NULL,
  status        TEXT NOT NULL,          -- 'in_progress' or 'done'
  response_code INTEGER,
  response_body TEXT,
  created_at    REAL NOT NULL
);
CREATE TABLE payments (id TEXT PRIMARY KEY, amount INTEGER, currency TEXT);
""")

def fingerprint(payload):
    canonical = json.dumps(payload, sort_keys=True, separators=(",", ":"))
    return hashlib.sha256(canonical.encode()).hexdigest()

def create_payment(key, payload):
    fp = fingerprint(payload)
    # Step 1: try to claim the key. Only one caller can insert it.
    cur = db.execute(
        "INSERT INTO idempotency_keys (key, fingerprint, status, created_at) "
        "VALUES (?, ?, 'in_progress', ?) ON CONFLICT(key) DO NOTHING",
        (key, fp, time.time()),
    )
    if cur.rowcount == 0:
        # Step 2: the key exists, so this is a retry (or a misuse).
        fp_old, status, code, body = db.execute(
            "SELECT fingerprint, status, response_code, response_body "
            "FROM idempotency_keys WHERE key = ?", (key,)).fetchone()
        if fp_old != fp:
            return 422, {"error": "key reused with a different request"}
        if status == "in_progress":
            return 409, {"error": "original request still in progress"}
        return code, json.loads(body)  # replay the stored response

    # Step 3: do the work and record the response in ONE transaction.
    db.execute("BEGIN")
    try:
        payment_id = "pay_" + uuid.uuid4().hex[:8]
        db.execute("INSERT INTO payments VALUES (?, ?, ?)",
                   (payment_id, payload["amount"], payload["currency"]))
        body = {"id": payment_id, "amount": payload["amount"]}
        db.execute(
            "UPDATE idempotency_keys SET status = 'done', response_code = 201, "
            "response_body = ? WHERE key = ?", (json.dumps(body), key))
        db.execute("COMMIT")
        return 201, body
    except Exception:
        db.execute("ROLLBACK")
        db.execute("DELETE FROM idempotency_keys WHERE key = ?", (key,))
        raise

key = str(uuid.uuid4())
first = create_payment(key, {"amount": 500, "currency": "usd"})
retry = create_payment(key, {"amount": 500, "currency": "usd"})
misuse = create_payment(key, {"amount": 900, "currency": "usd"})
print(first[0], retry[0], retry[1] == first[1], misuse[0])
print(db.execute("SELECT COUNT(*) FROM payments").fetchone()[0])

Running it prints 201 201 True 422 on the first line and 1 on the second: the retry got back the exact original response, the reused key with a different amount was rejected, and only one payment row exists. In production you would use PostgreSQL or another shared database instead of an in-memory one, but the shape stays the same: an atomic claim, a fingerprint check, and a single transaction that writes both the effect and the response.

Where real implementations break

Side effects outside your database

The example is safe because the payment and the response live in the same database transaction. Many real handlers call an external service, such as a card processor or an email provider, and that call cannot be rolled back. Brandur Leach's widely cited article on implementing Stripe-like idempotency keys in Postgres handles this by splitting the request into atomic phases separated by those foreign calls, recording a recovery point after each phase so a retry resumes where the last attempt stopped. When the external service supports idempotency itself, pass it a key derived from yours, so a crash mid-call cannot create a second charge downstream. Work that only needs to happen after success, such as a receipt email, can be staged as a job row in the same transaction and sent later.

Stuck in-progress keys

If the process crashes after claiming a key but before finishing, the row stays in progress forever and every retry gets a 409. The fix is a lock timestamp with a timeout: a retry may take over a key whose lock is older than some threshold. The same Postgres article also describes a background completer that finishes abandoned requests, and a reaper that deletes old keys and flags anything it cannot clean up for a human.

Key scope and retention

Scope keys to the caller, typically with a unique index on the account ID plus the key, so two customers who happen to generate the same string do not collide. Decide how long keys live and publish it, as the IETF draft recommends. Stripe's v1 window is at least 24 hours. The Postgres article suggests about 72 hours so that a bug found over a weekend can still be repaired and its requests completed. AWS keeps EC2 client tokens for the lifetime of the resource plus an extra interval.

Caching errors you should retry

Decide which outcomes are final. Stripe's v1 API does not save a result when the request fails validation or conflicts with a concurrent request, because endpoint execution never began, so those can be retried with the same key. A transient failure that happened before any side effect is usually better left unrecorded too; otherwise a retry replays the failure instead of attempting the work again.

A short production checklist

• Generate keys on the client, once per logical operation, and reuse the same key for every retry of that operation.
• Claim keys with a unique constraint, never with a read followed by a separate write.
• Store a request fingerprint and return 422 when it does not match.
• Return 409 while the original request is still running, and pair it with a lock timeout.
• Commit the side effect and the stored response atomically wherever possible.
• Pass derived keys to downstream services that support them.
• Scope keys per account, set a retention window, and document both.

Retries are unavoidable in distributed systems: networks drop connections, load balancers time out, and SDKs retry on their own. Idempotency keys do not prevent any of that. They make it harmless, which is the more useful property, as long as the server treats the key as a small state machine rather than a cache lookup.