Design a Payment System

Stripe / PayPal System Design

Overview

A payment system takes money from a buyer and credits a merchant — the backend behind a Stripe charge, a PayPal checkout, or the "Pay" button on any store. The user clicks once; behind that click the system must talk to an external payment service provider (PSP), move real money, and update the order — all without ever charging twice, losing a charge, or telling the customer "paid" when the money never left.

It is "Hard" because the core is correctness with money under failure. Networks time out, users double-click, and you cannot run a single database transaction across your own database and an external provider. So the design lives or dies on three things: charging exactly once even when the same request arrives twice (idempotency); keeping the money side and the order side consistent when a crash lands between them; and reconciling your own books against the PSP, which is the real source of truth for whether money moved. Get any of these wrong and you either lose money or charge a customer twice — both unacceptable.

In this walkthrough you'll scope it, size it, model a double-entry ledger, design the API, draw the authorize / capture / webhook flows, and go deep on idempotency, money/order consistency, and reconciliation.

Functional requirements

• Authorize a payment (place a hold). Check the card is valid and reserve the funds without taking them yet — like a hotel pre-authorizing a deposit.
• Capture a payment (take the money). Convert an authorization into an actual charge, in full or in part, when the merchant fulfills the order.
• Charge exactly once. A double-click, a client retry, or a network timeout must never result in two charges for one intended payment.
• Keep the order and the money in sync. The order is marked paid if and only if the money was actually captured — never one without the other.
• Refund a captured payment. Return money (in full or part) for a completed charge.

Out of scope (state it): fraud / risk ML (a separate scoring service), PCI compliance details (we assume card data is tokenized by the PSP and never touches our servers), and multi-currency FX (we charge in one currency). These are real but orthogonal; naming them keeps interview time on the money-correctness core.

Non-functional requirements

• Exactly-once effect on money. The headline. Repeating the same payment request must have the same effect as doing it once — no double charge, no double refund. This forces idempotency everywhere money moves.
• Strong consistency on the ledger. A payment's recorded state and the order's state must agree. We prefer to reject or retry a payment over recording an inconsistent one — the CP choice in CAP.
• Durability — never lose a committed money fact. Once we've recorded "captured," it must survive crashes. We never hold money state only in memory or only at the PSP.
• Auditability. Every money movement is traceable and immutable, because finance, disputes, and regulators all read these records. This pushes us toward an append-only ledger, not mutable balance rows.
• Reasonable throughput, modest QPS. Payments are far lower volume than reads on a social feed, but each one is precious. Optimize for correctness first, throughput second.

Estimations

State assumptions; the goal is to justify the ledger, the outbox, and the reconciliation job — not to be exact.

Rounded assumptions

• Payments: ~10M payments/day ⇒ ~100 payments/sec average, ~1K/sec at peak. Low QPS by web standards.
• Each payment produces several state transitions and ledger entries: authorize, capture, maybe refund ⇒ ~5–10 ledger rows per payment.
• Read traffic (merchants checking status, dashboards): ~10× the write rate, so ~1K reads/sec average.
• Retries: assume ~5–10% of requests are retried by clients or the network ⇒ idempotent dedup must handle on the order of tens of duplicate requests/sec (up to ~100/sec at peak).
• The PSP sends an asynchronous webhook per state change; assume ~1–5% of webhooks are delayed, duplicated, or lost — reconciliation must catch those.

How the numbers affect the design

Storage

• Per payment row: ids (~32 bytes), amount (8 bytes), currency (4 bytes), status (1 byte), timestamps (~16 bytes), PSP reference (~32 bytes) ≈ ~150 bytes. Plus ~5–10 ledger entries at ~64 bytes each.
• 10M payments/day × ~1 KB total ≈ ~10 GB/day, ~3.5 TB/year. Modest — it fits in a sharded relational store partitioned by payment or merchant. The data is small; the difficulty is keeping it correct, not storing it.

Core entities / data model

The model separates intent (what we're trying to do), money facts (the immutable ledger), and the order (owned by a different service). Keeping these apart is what lets each stay correct.

Payment.status is the heart of correctness: a state machine that only moves forward through legal transitions:

stateDiagram-v2
    [*] --> CREATED
    CREATED --> AUTHORIZED
    CREATED --> FAILED
    AUTHORIZED --> CAPTURED
    AUTHORIZED --> VOIDED: "authorization released,<br/>never captured"
    AUTHORIZED --> EXPIRED: "hold not captured<br/>in time (~7 days)"
    CAPTURED --> REFUNDED
    CAPTURED --> PARTIALLY_REFUNDED

The LedgerEntry rows are a double-entry ledger — every money movement writes two balanced rows (a debit and a credit on different accounts), so the books always sum to zero and a missing or duplicated entry is immediately visible. The ledger is append-only: you never update a balance in place; you post a new entry. Balances are derived by summing entries.

The Payment, LedgerEntry, IdempotencyKey, and OutboxEvent live in one strongly consistent relational database (Postgres/MySQL) so a payment write and its outbox event commit in the same local transaction (deep dive 2). The Order lives in a separate Order Service — that separation is exactly what makes consistency hard.

API design

A small REST API. Authorize and capture are deliberately separate so you can reserve funds at checkout and take them only on fulfillment. Every money-moving call carries an idempotency key.

Authorize a payment (place a hold)

POST /api/payments/authorize
Header: Idempotency-Key: <client-generated-uuid>
Body: { "order_id": "o_123", "amount": 4200, "currency": "usd", "payment_method_token": "tok_..." }
201 Created
Returns: { "payment_id": "pay_456", "status": "AUTHORIZED", "psp_ref": "ch_..." }   (replay returns the same body)

Validate the card with the PSP and reserve the funds without taking them — like a hotel placing a hold. The Idempotency-Key is mandatory: a double-click or network retry with the same key returns the original result instead of authorizing again (deep dive 1). amount is in the smallest currency unit (cents) to avoid floating-point money bugs.

Capture an authorized payment (take the money)

POST /api/payments/{payment_id}/capture
Header: Idempotency-Key: <client-generated-uuid>
Body: { "amount": 4200 }                 (amount ≤ authorized amount; omit for full capture)
200 OK
Returns: { "payment_id": "pay_456", "status": "CAPTURED", "captured_amount": 4200 }

Convert the hold into an actual charge when the merchant fulfills the order. Capture can be partial (ship two of three items, capture two-thirds). This is the step that triggers marking the order paid (deep dive 2). An authorization is not open-ended: the hold expires if not captured in time (commonly ~7 days, depending on the card network), so capture must happen within that window — otherwise you must re-authorize before you can take the money.

Void / release an authorization

POST /api/payments/{payment_id}/void
200 OK
Returns: { "payment_id": "pay_456", "status": "VOIDED" }

Release a hold that will never be captured (order cancelled before fulfillment), so the funds return to the customer. Legal only from AUTHORIZED.

Refund a captured payment

POST /api/payments/{payment_id}/refunds
Header: Idempotency-Key: <client-generated-uuid>
Body: { "amount": 4200 }                 (amount ≤ captured; omit for full refund)
201 Created
Returns: { "refund_id": "ref_789", "status": "PENDING" }   (settles to SUCCEEDED via webhook)

Return money for a completed charge. Refunds are themselves idempotent and asynchronous — the PSP confirms completion later via webhook.

Get payment status

GET /api/payments/{payment_id}
200 OK
Returns: { "payment_id": "pay_456", "status": "CAPTURED", "amount": 4200, "ledger": [ ... ] }

Read the current state and ledger entries. This is the read-heavy path (~10× writes), so it can be served from a read replica.

PSP webhook receiver (called by the PSP, not the client)

POST /api/webhooks/psp
Body: { "psp_event_id": "evt_...", "type": "charge.captured", "data": { "psp_ref": "ch_...", ... } }
200 OK

The PSP calls this asynchronously when a charge state changes. Because anyone on the internet can POST to this URL, the receiver must verify the PSP's signature — an HMAC over the raw payload using a shared secret, sent in a header — and reject the request before doing anything else if it doesn't match. The signed payload also includes a timestamp, so a replayed (otherwise-valid) webhook whose timestamp is outside a small tolerance is rejected — signature verification alone wouldn't stop a captured-and-replayed request. Only then is the event trusted. The psp_event_id is stored with a unique constraint so a re-delivered webhook is processed at most once (deep dive 2). This endpoint is how the system learns the real outcome of an async charge.

High-level architecture

Three flows matter: the authorize/capture write path (intent → PSP → ledger), the webhook path (the PSP tells us the real outcome asynchronously), and the reconciliation path (a daily job compares our ledger to the PSP's report). We'll walk each, then combine them.

1. Authorize / capture write path

A money-moving request first claims an idempotency key, then calls the PSP, then records the result and an outbox event in one transaction.

flowchart LR
    Client[Client] -->|"authorize/capture + Idempotency-Key"| PS[Payment Service]
    PS -->|"insert key (unique) or return stored result"| DB[(Payments DB)]
    PS -->|"charge card (idempotency key)"| PSP[(PSP)]
    PS -->|"in ONE tx: payment + ledger + outbox"| DB
    PS -->|"return result"| Client

1. The request arrives with a client-generated Idempotency-Key. The Payment Service tries to insert that key. If it already exists, this is a retry — return the stored result and stop (deep dive 1).
2. If the key is new, call the PSP to authorize or capture, passing the same key to the PSP so the PSP also dedups.
3. On the PSP response, write the Payment status, the double-entry LedgerEntry rows, the stored idempotency response, and an OutboxEvent ("payment captured") — all in one local database transaction.
4. Return the result to the client. The outbox event will later notify the Order Service (deep dive 2).

2. Webhook path (the PSP reports the real outcome)

Card charges are not always synchronous; the authoritative result often arrives later as a webhook.

flowchart LR
    PSP[(PSP)] -->|"POST webhook (psp_event_id)"| WH[Webhook Receiver]
    WH -->|"insert psp_event_id (unique) — dedup"| DB[(Payments DB)]
    WH -->|"advance state machine + post ledger"| DB
    WH -->|"emit outbox event"| DB
    WH -->|"200 OK (ack)"| PSP

1. The PSP POSTs a webhook with a unique psp_event_id whenever a charge changes state (captured, failed, refunded). The receiver first verifies the HMAC signature over the raw payload and drops the request if it fails — an unverified webhook is never trusted.
2. The receiver inserts the psp_event_id under a unique constraint. If it's already there, this is a duplicate delivery — ack and ignore (an idempotent consumer).
3. For a new event, advance the payment state machine and post any ledger entries in one transaction.
4. Return 200 OK so the PSP stops retrying. If you don't ack, the PSP re-delivers — which is exactly why the consumer must be idempotent.

3. Reconciliation path (the PSP is the source of truth)

Once a day the PSP publishes a settlement report; a batch job compares it to our ledger and flags mismatches.

flowchart LR
    PSP[(PSP)] -->|"daily settlement report"| Recon[Reconciliation Job]
    DB[(Payments DB)] -->|"our ledger for the day"| Recon
    Recon -->|"match by psp_ref; diff amounts/states"| Recon
    Recon -->|"auto-repair missed webhooks"| DB
    Recon -->|"flag true mismatches"| Alert[Ops / Exceptions Queue]

1. The PSP delivers a settlement report listing every charge it actually settled and the fees it took.
2. The job pulls our ledger for the same window and matches line by line on psp_ref.
3. Anything the PSP settled that our ledger doesn't show as captured is usually a missed webhook — the job repairs it by advancing our state (the same logic the webhook would have run).
4. Anything that still doesn't reconcile (we captured but the PSP shows nothing, or amounts differ) is a true mismatch — flag it to an exceptions queue for a human (deep dive 3).

4. Combined architecture

Putting it together — the Payment Service writes payment + ledger + outbox atomically, a relay delivers outbox events to the Order Service, the webhook receiver folds in async outcomes, and reconciliation backstops everything against the PSP.

flowchart LR
    Client[Client] --> GW[API Gateway]
    GW -->|"authorize/capture/refund"| PS[Payment Service]
    PS -->|"idempotency key + payment + ledger + outbox (one tx)"| DB[(Payments DB)]
    PS -->|"charge (idempotency key)"| PSP[(PSP)]
    Relay[Outbox Relay] -->|"read pending outbox events"| DB
    Relay -->|"publish 'payment captured'"| OS[Order Service]
    PSP -->|"webhook (psp_event_id)"| WH[Webhook Receiver]
    WH --> DB
    Recon[Reconciliation Job] -->|"compare ledger vs settlement report"| PSP
    Recon --> DB
    Recon -.->|"flag mismatches"| Alert[Exceptions Queue]

The Payments DB is the strongly consistent hub holding payments, the double-entry ledger, idempotency keys, and the outbox. The Outbox Relay reads committed outbox rows and reliably delivers them to the Order Service, so "money captured" and "order marked paid" stay consistent without a cross-system transaction (deep dive 2). The webhook receiver folds the PSP's async truth back in, and the daily reconciliation job is the safety net that catches whatever the live paths missed.

Deep dives

1. Idempotency — charging exactly once

The single most-probed part of this design: the same payment must result in one charge even when the request arrives two or more times. Duplicates are not rare edge cases — a double-click, a mobile network that times out and auto-retries, or a load balancer re-sending all create them constantly. The defense is an idempotency key: the client generates one unique key per intended payment and sends it on every retry of that payment. The server's job is to make the first request do the work and every retry replay the stored result.

The trap is the race. The naïve version is "read: does this key exist? If not, charge and store it." Two retries arriving at the same instant can both read "not found," both charge, and you've double-charged. This is a classic read-modify-write race, and you cannot fix it by reading more carefully — you fix it by making the claim atomic.

key = "idem-abc"  (amount $42)
  T1: SELECT key 'idem-abc' → not found
  T2: SELECT key 'idem-abc' → not found     ← both saw "absent"
  T1: charge PSP $42 → insert key
  T2: charge PSP $42 → insert key
  → customer charged $84. Double charge.

key = "idem-abc"  (amount $42), status column starts 'IN_PROGRESS'
  T1: INSERT key 'idem-abc' → succeeds (T1 owns this payment)
  T2: INSERT key 'idem-abc' → UNIQUE violation → this is a retry
  T1: charge PSP $42 → UPDATE key row: status=DONE, store response
  T2: sees key exists → wait for/read stored response → return it
  → customer charged $42 once; T2 replays the same result

2. Money/order consistency across services — outbox + saga

Capturing the card lives in the Payment Service and its database; marking the order paid lives in a separate Order Service. You want both or neither. But you cannot wrap a single database transaction around your database and an external PSP, and you don't want a two-phase commit across two services either — 2PC blocks whenever any participant is slow or down, which an external PSP frequently is.

The failure to avoid is "charge the card, then call the Order Service to mark it paid." If the process crashes between those two steps, the money moved but the order never updates — the customer paid and got nothing. Retrying the whole thing risks charging again. You need the "money moved" fact and the "tell the order" intent to be recorded together, atomically, durably, and delivered afterward with retries.

The canonical pattern is the transactional outbox combined with a saga:

flowchart LR
    PS[Payment Service] -->|"1. capture (idempotency key)"| PSP[(PSP)]
    PS -->|"2. one tx: payment=CAPTURED + ledger + outbox row"| DB[(Payments DB)]
    Relay[Outbox Relay] -->|"3. poll pending outbox"| DB
    Relay -->|"4. publish 'payment.captured'"| OS[Order Service]
    OS -->|"5. mark order PAID (dedup by payment_id)"| OSDB[(Order DB)]

1. The Payment Service captures via the PSP (idempotent, deep dive 1).
2. In one local transaction, it writes Payment=CAPTURED, the double-entry ledger rows, and an OutboxEvent row (payment.captured). Because the event commits in the same transaction as the money fact, it cannot be lost — if the transaction commits, the event is there; if it rolls back, neither happened.
3. A separate Outbox Relay polls for pending outbox rows (or tails the database change log).
4. It publishes payment.captured to the Order Service, retrying until acked, then marks the outbox row sent.
5. The Order Service marks the order PAID and dedups by event_id (not payment_id alone — one payment legitimately emits several distinct events: capture, partial refund, chargeback) — an idempotent consumer, because the relay may deliver the same event more than once (at-least-once delivery).

This gives exactly-once effect even though delivery is at-least-once: the event is never lost (committed with the data) and never double-applied (the consumer dedups). If a later saga step fails — say the order can't be fulfilled after capture — you run a compensating action: issue a refund (itself idempotent) and move the payment to REFUNDED, rather than trying to "undo" a committed transaction.

3. Reconciliation and settlement — the PSP is the source of truth

Your ledger is what you believe happened. The PSP's records are what actually happened to the money. These can drift: a webhook is lost so you never recorded a capture; a webhook is duplicated; a charge succeeds at the PSP but your commit failed; fees differ from what you estimated. Reconciliation is the daily batch job that compares the two and forces them back into agreement — and it's why the system is trustworthy despite every live path being best-effort.

Settlement vs authorization. Capturing a payment doesn't instantly put cash in the merchant's bank. The PSP batches captured charges and settles them — moves the actual funds, minus fees — typically on a daily cycle, then publishes a settlement report. So a payment is CAPTURED in your ledger (you've taken it) well before it is settled (money actually transferred). A reasonable default is to mark the order PAID on capture (funds authorized and captured) rather than waiting on settlement — capture is the point at which the customer's money is committed, so it's the natural trigger for fulfillment. It isn't the only right answer: some businesses gate fulfillment on settlement or a risk decision (e.g. high-fraud goods, thin margins) and treat capture as merely "pending." Whichever point you pick, a later settlement failure or reversal is rare and is handled as a compensating action (reverse the charge, move the payment to REFUNDED), not by holding the order in limbo until settlement clears. The ledger models the timing with separate accounts/states — pending (captured, awaiting settlement) vs settled — so you always know how much is real cash vs in-flight.

How reconciliation runs:

For each line in the PSP settlement report (matched by psp_ref):
  1. Find our ledger entry for that psp_ref.
  2. PSP settled, we show CAPTURED+pending → mark settled, post fee entries.   (normal)
  3. PSP settled, we have NO capture        → MISSED WEBHOOK: replay the
                                               capture (advance state machine,
                                               post ledger) — auto-repaired.
  4. We show CAPTURED, PSP shows nothing     → mismatch ONLY if past the
                                               expected settlement window;
                                               before that it's normal pending
                                               (a late capture settles next
                                               day). Past the window → flag ops.
  5. Amount/fee differs from expected        → flag for review.

Case 3 is the important one: a missed webhook doesn't lose money, because the daily reconciliation against the PSP's report catches the gap and repairs it using the same idempotent state-machine logic the webhook would have run. That is what lets the live webhook path be best-effort — reconciliation is the durable backstop. Cases 4 (once past the settlement window) and 5 are genuine exceptions that need a human; they go to an exceptions queue, ideally near-zero in volume.

Because the ledger is double-entry, reconciliation has a second, cheaper check it runs continuously: the books must balance. Every payment's debits and credits must sum to zero; if they don't, an entry was dropped or duplicated, and you've found a bug without even consulting the PSP.

Tradeoffs & bottlenecks

• Exactly-once effect, not exactly-once delivery. You cannot guarantee a message or webhook is delivered exactly once. You make the effect exactly-once by combining at-least-once delivery (retries) with idempotent consumers (dedup). Naming this distinction is the core insight.
• Consistency over availability. For money you choose CP: better to fail a payment and let the client retry than to record an inconsistent or duplicated charge. The read path (status, dashboards) can be AP and served from replicas.
• No 2PC across DB + PSP. A distributed transaction across your database and an external provider would block under the provider's failures. The outbox + saga + reconciliation pattern trades atomic simplicity for resilience — the right trade when one participant is an external service.
• Authorize-then-capture vs charge directly. Splitting auth and capture supports holds, partial captures, and cancel-before-fulfillment, at the cost of more states and a void path. A single immediate charge is simpler but can't model fulfillment delays.
• Idempotency key TTL. Too short and a late retry double-charges; too long and the key table grows and a genuinely new operation could collide with an old key. Tune to cover the longest realistic retry window (a day or two).
• Outbox relay lag. The outbox decouples money from order updates, but the relay polls, so "order marked paid" trails "money captured" by seconds. That's acceptable — the customer sees "payment processing" briefly — and far safer than a synchronous cross-service call that can lose money on a crash.
• Ledger as append-only. Append-only gives auditability and makes errors visible, but balances must be derived (summed) rather than read from a single row. Cache or snapshot balances if the read path needs them hot.

Extensions if asked

If the interviewer wants to push beyond the core, add only the extension that changes the design discussion. (These are standalone topics; there's no dedicated guide for them yet.)

• Refunds and chargebacks/disputes. Refunds reuse the idempotent, async, webhook-confirmed flow. Chargebacks are PSP-initiated reversals that arrive as webhooks and post compensating ledger entries — a money movement you don't control, which makes reconciliation even more important.
• Multi-currency and FX. Charging and settling in different currencies adds exchange-rate capture at authorize time, rounding rules, and currency-tagged ledger accounts. Out of scope here, but it changes the ledger model.
• Payouts to merchants. The other side of the flow: aggregating settled funds and paying merchants on a schedule, with its own ledger accounts and reconciliation against the bank.
• Fraud and risk scoring. A scoring service that can decline or hold an authorization before capture. It plugs in at authorize time but is its own ML subsystem.
• PCI scope reduction / tokenization. How card data is tokenized by the PSP so raw card numbers never touch your servers, shrinking your compliance scope. We assumed tokenization throughout.

What interviewers look for & common mistakes

What interviewers usually reward:

• An atomic idempotency mechanism — an insert-or-return on a unique key, not a check-then-act — so concurrent retries charge exactly once, plus a sensible key TTL and request-fingerprint check.
• A transactional outbox + saga to keep money and order consistent without 2PC, with the event committed in the same transaction as the money fact and an idempotent consumer on the other side.
• Reconciliation against the PSP as the source of truth, catching missed/duplicated webhooks and repairing them, backed by a double-entry, append-only ledger that must always balance.
• Authorize-then-capture modeled as a forward-only state machine, with void and refund as explicit transitions.
• Explicitly choosing consistency over availability for money while keeping reads on replicas.

Before you finish, do a quick mistake check:

• Did you make the idempotency claim atomic (unique-constraint insert), not a racy read-then-charge?
• Did the client generate the idempotency key (one per intended payment), with a TTL covering all retries?
• Did you avoid "charge then update the order" as two unguarded calls, using an outbox so the event commits with the money fact?
• Did you make webhook and event consumers idempotent (dedup by psp_event_id / event_id, not payment_id alone — one payment emits several distinct events)?
• Did you treat the PSP as the source of truth and add a daily reconciliation job that repairs missed webhooks?
• Did you use a double-entry, append-only ledger and distinguish pending (captured) from settled?
• Did you choose consistency over availability for money writes?