Design a Delayed Payment Scheduler

Scheduled Robux Transfer System Design

Overview

A delayed payment scheduler lets a user schedule a virtual-currency transfer to happen later. For example, a user may schedule a transfer of 500 Robux from account A to account B next Friday at noon. The user should also be able to cancel the transfer before it runs and later check whether it succeeded or failed.

The user-facing action looks simple. The hard part is moving money correctly across time and failures. The system must handle several important cases:

1. A scheduled payment must survive crashes. We cannot just keep it in memory because the payment may be scheduled days or weeks in the future.
2. The money must be moved exactly once, even if a worker crashes in the middle of execution and another worker retries the same payment.
3. The system must also stay reliable when many payments become due at the same time, such as midnight, the top of the hour, or the start of a promotion.

In this walkthrough, we will cover the scope of the system, estimate the scale, define the payment data model and state machine, design the APIs, and explain the create and execute flows. Then we will go deeper into durable scheduling, retry safety, exactly-once execution, and handling traffic spikes when many payments are due at once.

Functional requirements

For this system, we would focus on the core user actions and the correctness guarantees around execution. The main functional requirements are:

1. User can schedule a payment. A user or internal service can create a future transfer from account A to account B with an amount and an execute_at time. Once the system accepts the request, the schedule must be saved durably.
2. User can cancel the payment. The creator can cancel the payment while it is still pending. If execution has already started, the cancellation should fail clearly.
3. User can check payment status. Authorized users should be able to check the current state of a payment, such as scheduled, executing, executed, failed, or cancelled.

A few things we'll deliberately leave out of scope:

1. Users setup recurring payments.
2. Cross-currency conversion.

Non-functional requirements

Before going into the design, we would call out the main non-functional requirements. For this problem, correctness is more important than low latency.

1. Correctness / Exactly-once execution. A scheduled payment should never be executed twice and should never be silently lost. Each payment should either execute exactly once or end in a clear terminal state, such as FAILED or CANCELLED.
2. Durability. Once the system accepts a scheduled payment, it must be stored durably. It cannot live only in memory or in an in-process timer, because the payment may be scheduled days or weeks in the future.
3. Consistency between scheduler and ledger. The scheduled payment state and the ledger transfer result must agree. We should not mark a payment as EXECUTED unless the money was actually moved. If the transfer cannot be completed safely, it is better to mark the payment as FAILED than to record an inconsistent result.
4. Bounded scheduling delay. Payments should fire within a small, bounded delay of their due time (seconds, sometimes a minute under load). Money movement tolerates a little lateness; it does not tolerate being lost or duplicated.
5. Scalability under bursts. Many payments may become due at the same time, such as midnight, the top of the hour, or during a promotion. The system should handle these bursts by adding more workers, instead of depending on a single worker to process all due payments.

Estimations

We will do a quick back-of-the-envelope estimation here. The exact numbers are not important. The goal is to understand the scale and justify the design choices, such as a durable due-time index, leased workers, and sharded buckets.

Rough assumptions

1. Payments scheduled per day：Assume around 10M scheduled payments per day. That is about 115 new scheduled payments per second on average. Even if peak traffic is a few times higher, the create path is still relatively small.
2. Pending payments: Users may schedule payments days or weeks in advance. So the system may need to store a large number of pending payments. Assume around 500M pending scheduled payments at any given time.
3. Hot due time: Some events may cause many payments to become due at the exact same time. For example, a promotion unlock or a "pay everyone at midnight" batch may create a large spike. Assume around 5M payments become due at 00:00 and need to be processed within a bounded time window
4. Execution throughput: If we need to process 5M due payments within about one minute, the system needs roughly 5M / 60 seconds ≈ 83K executions per second. This is much higher than the normal create rate, so the execution path needs to be highly scalable.
5. Status reads: Status reads are probably moderate. Users, internal services, and dashboards may check payment state.Assume status reads are around 10x the create rate, which is still manageable compared with the hot execution path.

How the estimates affect the design

The numbers are not meant to be exact. They help us understand where the real difficulty is and what the design should optimize for.

Storage. A scheduled-payment row: ids (~32 bytes), amount (8 bytes), execute_at (8 bytes), status (1 byte), idempotency/version (~24 bytes), timestamps (~16 bytes) ≈ ~100–150 bytes. 500M pending × ~150 bytes ≈ ~75 GB live — small, but only if it's a sharded store partitioned by due-time so no single node holds or scans the whole backlog.

Core entities / data model

Two things are stored: the scheduled payment (intent + a state machine) and the ledger it writes to on execution. The money itself lives in a double-entry ledger; the scheduler orchestrates when a movement is applied, not the balances themselves.

ScheduledPayment.status is the heart of correctness — a state machine that only moves through legal transitions:

stateDiagram-v2
    [*] --> SCHEDULED
    SCHEDULED --> EXECUTING
    EXECUTING --> EXECUTED
    EXECUTING --> FAILED: insufficient funds / permanent error
    SCHEDULED --> CANCELLED: only from SCHEDULED,<br/>before execution starts
    EXECUTED --> [*]
    FAILED --> [*]
    CANCELLED --> [*]

Two rules make this safe. Cancel is a compare-and-set from SCHEDULED to CANCELLED: if a worker has already flipped the row to EXECUTING, the cancel's CAS fails and is rejected — you cannot cancel a payment that is mid-flight. Execution is a CAS from SCHEDULED to EXECUTING: only one worker can win that transition, which is the first line of defense against double-dispatch (deep dive 2). Both cancel and execute race on the same SCHEDULED precondition, so exactly one wins: whichever CAS commits first flips the row, and the other's CAS finds the precondition no longer holds and is rejected — a cancel that arrives a moment too late fails cleanly (returns 409) and a worker that reads a row just before it's cancelled fails its execute CAS and skips it. The version column supports optimistic concurrency for these transitions.

API design

A small REST API. Every mutating call carries an idempotency key so a retry of the scheduling request never creates two payments.

Schedule a payment

POST /api/payments
Header: Idempotency-Key: <client-generated-uuid>
Body: { "from_account": "a_1", "to_account": "a_2", "amount": 500, "execute_at": "2026-07-03T12:00:00Z" }
201 Created
Returns: { "payment_id": "pay_789", "status": "SCHEDULED", "execute_at": "2026-07-03T12:00:00Z" }   (replay returns the same body)

Record the transfer to run in the future. The server inserts the idempotency key atomically (deep dive 2's dedup pattern), writes the ScheduledPayment as SCHEDULED, and places it in the due-index bucket for its execute_at. amount is in whole Robux (an integer) to avoid floating-point money bugs. Validation of the amount and balance happens at execution time, not now — the sender's balance a week from now is what matters (deep dive 2).

Cancel a scheduled payment

DELETE /api/payments/{payment_id}
200 OK
Returns: { "payment_id": "pay_789", "status": "CANCELLED" }
409 Conflict   (if already EXECUTING / EXECUTED — too late to cancel)

Cancel a still-pending payment. Implemented as a compare-and-set from SCHEDULED to CANCELLED. Cancel is idempotent, but that doesn't fall out of the CAS for free: a second cancel fails the SCHEDULED precondition exactly like a too-late cancel does, so on a failed CAS the handler reads the row's current status and branches — already CANCELLED returns 200 (the idempotent repeat), while EXECUTING or a terminal state returns 409 (too late, the payment already ran or is running). Without that post-failure status read, a repeat cancel would look identical to a too-late one.

Get payment status

GET /api/payments/{payment_id}
200 OK
Returns: { "payment_id": "pay_789", "status": "EXECUTED", "amount": 500, "executed_at": "2026-07-03T12:00:03Z" }

Read the current state. This is the read-heavy path and can be served from a read replica. A FAILED payment includes a reason (e.g. insufficient_funds) so the caller knows why it didn't move.

List a caller's scheduled payments

GET /api/accounts/{account_id}/payments?status=SCHEDULED&cursor=<opaque>&limit=20
200 OK
Returns: { "items": [ { "payment_id": "...", "execute_at": "...", "status": "..." }, ... ], "next_cursor": "<opaque>" }

List a user's upcoming (or past) payments, paginated with an opaque cursor. Backed by a secondary index on from_account, separate from the due-index that drives execution.

High-level architecture

Two paths matter: a create path that durably records a scheduled payment, and an execution path where workers find due payments and move the money exactly once. We'll walk each, then combine them.

1. Create path (schedule a payment)

Scheduling is a durable write: dedup the request, store the payment, and index it by due-time.

flowchart LR
    Client[Client / Internal Service] -->|"POST payment + Idempotency-Key"| SS[Scheduler Service]
    SS -->|"insert key (unique) or return stored result"| DB[(Scheduled Payments DB)]
    SS -->|"write payment=SCHEDULED + due-index row (one tx)"| DB
    SS -->|"return payment_id + status"| Client

1. The request arrives with a client-generated Idempotency-Key. The Scheduler Service tries to insert it. If it already exists, this is a retry — return the stored result and stop.
2. If the key is new, write the ScheduledPayment (status SCHEDULED) and its DueIndex row (bucketed by execute_at) in one local transaction, so a payment is never stored without being findable, and never indexed without being stored.
3. Return the result. The payment is now durable and will be picked up when its bucket comes due — no in-memory timer is involved.

2. Execution path (fire due payments exactly once)

Workers lease a slice of the due-index, claim each due row via a state transition, and apply the transfer idempotently.

flowchart LR
    W[Dispatcher Worker] -->|"1. lease a due bucket-shard"| DB[(Scheduled Payments DB)]
    W -->|"2. read batch where due_at <= now"| DB
    W -->|"3. CAS SCHEDULED -> EXECUTING"| DB
    W -->|"4. apply transfer (idempotent double-entry txn)"| Ledger[(Ledger)]
    W -->|"5. CAS EXECUTING -> EXECUTED (or FAILED)"| DB

1. A Dispatcher Worker claims a lease on a due bucket-shard (deep dive 1) so two workers don't drain the same slice at once.
2. It reads a batch of rows in that slice whose due_at (the payment's execute_at) <= now, judged against an authoritative server time, not the worker's wall clock. The claim query selects both fresh work and stranded work: status = SCHEDULED OR (status = EXECUTING AND lease_expiry < now()) — so a row a dead worker left in EXECUTING is re-selected once its lease lapses, instead of draining only SCHEDULED rows and silently dropping the stranded one.
3. For each, it does a single-winner compare-and-set that stamps the new owner + expiry, with expected state either SCHEDULED or EXECUTING-with-an-expired-lease. Only one worker can win that takeover transition — two workers can't both "recover" the same expired row, so recovery is itself single-winner (the money is already safe via idempotency, but the takeover must have exactly one winner). A row already EXECUTING with a live lease, or EXECUTED / CANCELLED, fails the CAS and is skipped.
4. It applies the transfer as a single atomic double-entry ledger transaction. The payment_id guard is checked first: if the payment was already applied, the apply short-circuits (no balance re-check, no second movement). Otherwise it checks the balance and posts the debit/credit (deep dive 2).
5. It CAS-es the row to EXECUTED — or to FAILED only when a genuine first apply's balance check failed and no money moved. A crash before step 5 is safe: the payment is still EXECUTING, its lease expires, another worker re-claims it, sees the payment_id already applied, and converges the row to EXECUTED — a retry must never turn already-moved money into a FAILED.

3. Combined architecture

Putting it together — the create path durably records and indexes payments; a fleet of leased workers drains the due-index in parallel and applies transfers idempotently; a monitor re-queues stuck rows.

flowchart LR
    Client[Client / Internal Service] --> GW[API Gateway]
    GW -->|"schedule / cancel / get"| SS[Scheduler Service]
    SS -->|"payment + due-index + idempotency key (one tx)"| DB[(Scheduled Payments DB)]
    Workers[Dispatcher Workers - leased per bucket-shard] -->|"claim due rows (CAS)"| DB
    Workers -->|"apply transfer (idempotent, guarded by payment_id)"| Ledger[(Double-entry Ledger)]
    Workers -->|"CAS EXECUTED / FAILED"| DB
    Monitor[Lease Monitor] -.->|"re-queue expired-lease / stuck EXECUTING rows"| DB
    GW -->|"status (read replica)"| Replica[(Read Replica)]

The Scheduled Payments DB is the durable hub holding payments, the due-index, and idempotency keys. The Dispatcher Workers scale horizontally, each leasing bucket-shards so the same payment is never drained twice. The Lease Monitor backstops crashes by making expired-EXECUTING rows re-claimable: primarily, the drain query already selects status = SCHEDULED OR (status = EXECUTING AND lease_expiry < now()), so a stranded row is picked up directly once its lease lapses; alternatively the monitor resets expired EXECUTING rows back to SCHEDULED. Either way a row stuck in EXECUTING past its lease becomes re-eligible, and a worker retries it safely because the ledger apply is idempotent. Status reads go to a replica so dashboards never compete with the execution path.

Deep dives

1. Durable scheduling under failure — how "run it later" survives a week of crashes

The defining constraint: a payment scheduled for next Friday must still fire if every server restarts twice before then. That rules out holding the schedule in memory. The question is where the schedule lives and how workers find what's due — here are three approaches, worst to best.

  Mon: schedule 200 payments → 200 in-process timers armed
  Thu: deploy / crash → process restarts → ALL 200 timers gone
  Fri: nothing fires. 200 payments silently lost.

  every 1s:  SELECT * FROM payments
             WHERE due_at <= now() AND status = 'SCHEDULED'
             ORDER BY due_at LIMIT 500
  → execute the batch, mark EXECUTED

flowchart LR
    Bucket["Bucket 2026-07-03T12:00<br/>5M due, split into 64 shards"] --> S0["shard 0"]
    Bucket --> S1["shard 1"]
    Bucket --> Sdots["..."]
    Bucket --> S63["shard 63"]
    S0 -->|"lease"| WA["worker A<br/>drains ~78K rows"]
    S1 -->|"lease"| WB["worker B<br/>drains ~78K rows"]
    Sdots -->|"lease"| Wdots["...<br/>(5M / 64 ≈ 78K each)"]
    S63 -->|"lease"| WZ["worker Z<br/>drains ~78K rows"]
    WA --> Drain["64 workers drain the minute in parallel;<br/>add workers to go faster"]
    WB --> Drain
    Wdots --> Drain
    WZ --> Drain
    style Bucket fill:#dcfce7,stroke:#16a34a,color:#14532d

2. Exactly-once execution despite retries — the money moves once

A due payment will be dispatched at least once — and under crashes, sometimes more than once: a worker claims a row, moves the money, then dies before marking it EXECUTED; its lease expires; another worker re-claims the same row. You cannot prevent every duplicate dispatch, so the money movement itself must be safe to repeat. The goal is exactly-once effect, achieved by at-least-once dispatch plus an idempotent apply.

The apply is an idempotent, atomic ledger transaction. Moving 500 Robux from A to B is a single database transaction that (1) writes the debit and credit double-entry rows and (2) records that payment_id has been applied — guarded by a unique constraint on payment_id in the ledger (or an applied-payments table). If a second worker retries the same payment, the unique constraint rejects the duplicate apply: the transaction is a no-op, the money is not moved again. Correctness rests on this, not on any lock.

retry after a worker crashed mid-execution:
  worker A: CAS SCHEDULED→EXECUTING → apply ledger txn (payment_id pay_789) → CRASH before EXECUTED
  lease expires; worker B re-claims pay_789 (still EXECUTING)
  worker B: apply ledger txn (payment_id pay_789)
            → UNIQUE(payment_id) violation → apply is a NO-OP
  worker B: CAS EXECUTING→EXECUTED
  → money moved exactly once; state converges to EXECUTED

A lease/lock reduces duplicates but does not provide correctness. A distributed lock on the bucket-shard, plus the CAS to EXECUTING, means most of the time exactly one worker touches a payment. But a lock can betray you: the lease can expire while a worker is paused in a long garbage-collection pause, or clocks can skew, and now two workers both believe they hold it. That is exactly why the idempotent ledger apply is mandatory — the lock is an optimization that avoids wasted duplicate work; the unique-constraint apply is the guarantee that duplicate work is harmless.

  worker A: acquire lease (TTL 30s) → begin transfer
  worker A: 35s GC pause  ← lease expired at 30s
  worker B: lease free → acquire → transfer AGAIN
  → without an idempotent apply, the money moves twice

  worker A: INSERT applied(pay_789) + debit A + credit B  → commits
  worker B: INSERT applied(pay_789) ...  → UNIQUE violation → rollback (no-op)
  → money moved once regardless of how many workers ran it

Balance is checked at execution, not scheduling — but only on a genuine first apply. The sender's balance a week from now is what matters, so the funds check happens inside the apply transaction. The subtlety: the payment_id guard must be checked before the balance. If the payment was already applied (the unique record exists), the worker converges the row to EXECUTED and never re-checks the balance. Skip this ordering and a worker retrying after a crash could re-read a now-insufficient balance and mark FAILED a payment whose money already moved — a payment shown as failed that actually executed. So FAILED is reachable only when a first apply's balance check fails before any debit/credit is committed (or on a permanent error), and it is not retried forever. Transient errors (a brief store outage) are retried with backoff.

3. Scheduler accuracy at scale — the hot wall-clock minute (the headline)

Real schedules cluster. Everyone picks "midnight," "top of the hour," or the exact second a promo unlocks, so millions of payments share one due bucket and come due together — a thundering herd. A design that fires them one poller at a time falls minutes or hours behind. Four techniques keep the scheduler accurate under the herd.

Shard the hot bucket (the primary fix). As in deep dive 1, the 12:00 bucket is split into N shards drained by N workers in parallel. The hotter the bucket, the higher you set N. This is what turns a 5M-payment spike from an impossible single-node drain into ~78K rows per worker across 64 workers. One subtlety: a payment's shard is fixed at schedule time — a week early, via hash(payment_id) % N — but a bucket's hotness isn't known until near its due time, so you can't retroactively "raise N for a hot bucket" over rows already hashed into a smaller shard space. Choose N at schedule time from a forecast of the bucket's load, or sub-divide shards dynamically as the due time approaches; you cannot re-shard rows already placed.

Batch-claim per poll. Each worker claims a batch (say 500 rows) per poll and executes them, rather than one row per round trip. This amortizes the poll cost and keeps throughput high without hammering the store with tiny queries.

Add jitter where exact timing isn't required. If a payment only needs to fire "around midnight," add a small random offset (e.g. ±30s) at schedule time, so the 5M payments come due smoothly across a ~60-second window instead of at a single instant — the workers see a steady stream rather than one spike. Jitter is opt-in: a payment that must fire at 12:00:00 keeps its exact time and relies on sharding instead.

Backpressure and catch-up, never drop. If the fleet still falls behind, payments simply wait in their durable bucket and are drained oldest-first as capacity frees up. Durability guarantees nothing is lost — a payment fires late, not never. The accuracy target is a bounded delay (fire within seconds, occasionally a minute under a severe spike), not exact-instant firing. Monitor the oldest un-drained bucket as the lag signal and autoscale workers on it.

Use an authoritative clock, not each worker's wall clock. "Is this payment due?" must be judged against a server-authoritative time (the store's now() or a time service), not each worker's local wall clock, which can skew arbitrarily and fire payments early or skip them. Note that in a sharded store there is no single global now() — but that's fine: the seconds-scale accuracy target makes millisecond skew between shard primaries irrelevant (assuming NTP-bounded skew of tens of ms; an unsynchronized primary could drift by seconds, which would matter at a seconds-scale target), so each shard judging its own rows against its own primary's clock is enough. You don't need a global clock; you only need to stop trusting the worker's clock.

5M payments all due at 12:00:00 exactly:

  one poller, ~2K/s:                 5M / 2K ≈ 42 min behind        ✗
  64 shards × 64 workers, ~1.3K/s ea: ~83K/s aggregate → ~60s drain ✓
  + opt-in ±30s jitter:               load arrives smoothly across a
                                      minute instead of at one instant ✓

The ~1.3K/s per worker above is illustrative, not a given. Each execution is an ACID double-entry ledger transaction — a unique-constraint insert on payment_id plus status-CAS round-trips — so a single worker's real execution rate is bounded by that transaction's latency, realistically far below 1.3K/s. Size N (the shard count) and the worker count from your measured ledger-transaction throughput, not the illustrative figure: if a worker sustains only a few hundred executions/sec, you need proportionally more shards and workers to drain the same 5M-payment minute.

Tradeoffs & bottlenecks

• Exactly-once effect, not exactly-once dispatch. You can't dispatch a due payment exactly once under crashes; you make the effect exactly-once with at-least-once dispatch + an idempotent, payment_id-guarded ledger apply. Naming this distinction is the core insight.
• Durable store vs in-memory timers. A database-backed due-index costs polling and indexing overhead but is the only thing that survives a week of restarts. In-memory timers are simpler and more precise but lose everything on a crash — unacceptable for money.
• Accuracy vs load. A shorter poll interval and no jitter fire payments closer to their exact time but increase store load and herd severity. A bounded delay (seconds) with jitter where allowed is the practical trade.
• Lease as optimization vs guarantee. A distributed lock removes most duplicate work but can double-grant under GC pauses and clock skew. It must never be the correctness mechanism — the idempotent apply is.
• Shard count N. Higher N drains hot buckets faster but adds coordination and many small slices for cold buckets. Tune N per bucket to its expected due-count.
• Cancel vs execution race. Modeling cancel as a CAS from SCHEDULED means a cancel that arrives after execution starts cleanly loses (returns 409) rather than corrupting a payment mid-flight — correctness over "always let the user cancel."

Extensions if asked

If the interviewer wants to push beyond the core, add only the extension that changes the design discussion. (These are standalone topics; there's no dedicated guide for them yet.)

• Recurring / repeating schedules. "Every Friday" means, on execution, computing the next execute_at and scheduling a fresh payment — turning a one-shot into a chain, with care not to duplicate on retry.
• Reschedule / edit before execution. Changing the time is a CAS from SCHEDULED that moves the row to a new due bucket, subject to the same "too late if EXECUTING" rule as cancel.
• Notifications on execution. Emitting a "payment sent" event to a separate notification system, fed by the same at-least-once event stream with an idempotent consumer.
• Fairness across senders. Preventing one account's million scheduled payments from starving others in the same hot bucket — per-sender rate limiting or weighted draining.
• Exactly-when guarantees. If a payment must fire at a precise instant (not a bounded delay), a dedicated timer-wheel with tighter polling and reserved capacity for that bucket.

What interviewers look for & common mistakes

What interviewers usually reward:

• A durable, indexed, shardable schedule — payments as rows in a time-bucketed due-index, never in-process timers — drained by many leased workers.
• Exactly-once effect via at-least-once dispatch + an idempotent ledger apply guarded by payment_id, with the explicit point that a lock is only an optimization.
• Handling the hot wall-clock minute — sharding the bucket, batch-claiming, jitter where allowed, and oldest-first backpressure that fires late but never drops.
• A forward-only state machine with cancel and execution modeled as compare-and-set transitions that race safely.
• Checking the balance at execution time and failing fast into a terminal FAILED state, not retrying a permanent error forever.

Before you finish, do a quick mistake check:

• Did you store payments durably (rows in a due-index), not in in-memory timers that a restart erases?
• Did you shard the due buckets so a hot minute is drained by many workers, not one poller?
• Did you make the money apply idempotent (unique payment_id), so a retried/duplicate dispatch is a no-op?
• Did you say the lease/lock is only an optimization, with idempotency as the real guarantee?
• Did you model cancel and execute as compare-and-set transitions so a late cancel loses cleanly?
• Did you check the balance at execution time and fail permanent errors into a terminal state instead of retrying forever?
• Did you judge due-ness against one authoritative clock, not each worker's wall clock?