Design a Notification System

Push / Email / SMS System Design

Overview

A notification system is a shared service other systems call to reach users across channels — mobile push, email, SMS, and in-app. When the payments service wants to send a receipt, the social service wants to say "Alex liked your photo," or the security service wants to send a login code, none of them should each build their own delivery plumbing. They call one notification service that accepts the request, picks the right channel(s), respects the user's preferences, talks to the external providers, retries on failure, and tracks whether the message actually arrived.

It is labeled "Medium" because the request looks trivial ("just send a message") but the hard parts are about scale and reliability: one event can fan out to millions of users at once; delivery must be reliable even when an external provider is slow or down; retried requests must not create duplicate work; and per-user limits must stop a user from being spammed. A strong answer leans on queues to decouple stages, per-channel workers, at-least-once delivery with idempotency, and throttling.

In this walkthrough you'll scope it, size it, model the data, design the producer-facing API, draw the ingestion and delivery paths, and deep-dive the multi-channel delivery pipeline, idempotency and fan-out, and user preferences and throttling.

Functional requirements

• Accept a notification request. A producer (another internal service) sends a request: who to notify, which template, which channel(s), and any data to fill the template with.
• Deliver across channels. Send via mobile push, email, SMS, and/or in-app, using the right external provider for each.
• Honor user preferences and opt-out. Respect which channels a user has enabled and never deliver to a user who opted out of that category.
• Support templates. Producers reference a named template ("password_reset", "price_drop") rather than sending raw, pre-rendered text.
• Track delivery status. Record whether each attempt was accepted, delivered, bounced, or permanently failed.

Out of scope (state it): the business logic that decides when to notify (that lives in the calling services — this system only delivers what it's told to send), deep delivery analytics/reporting, and the UI for editing preferences. We keep delivery, fan-out, reliability, and throttling central — they are the whole problem.

Non-functional requirements

• High throughput and fan-out. Handle millions of notifications per day at steady state, and absorb a single event that fans out to millions of users without collapsing.
• Reliable, at-least-once delivery. A notification that was accepted should eventually be delivered (or recorded as a permanent failure), even across worker crashes and provider hiccups.
• Minimize duplicates the user can see. At-least-once delivery and provider retries can create duplicates; the system uses idempotency and provider-specific dedup features where available, but external sends can never be made perfectly exactly-once.
• Per-user throttling. Cap how many notifications a single user receives in a window so a misbehaving producer or a busy day can't spam them.
• Failure isolation. A slow or failing external provider on one channel (e.g. SMS) must not stall the others (push, email).

Estimations

State assumptions; the aim is to justify the queue-based, per-channel design.

Rounded assumptions

• Notifications sent per day: ~1B across all channels.
• That averages ~1B / 86,400 ≈ ~12K notifications/sec, but traffic is bursty — call peak ~5×, so ~60K/sec.
• Channel split (rough): 70% push, 20% email, 8% in-app, 2% SMS. Push dominates volume; SMS is low-volume but expensive per message.
• Fan-out spike: one broadcast event ("your team scored," a breaking-news alert) targets ~10M users at once. Even spread over a minute that's ~170K sends/sec for that one event — several times peak steady-state, from a single request.
• Average payload is small (a few hundred bytes of template data + addresses), so storage of in-flight requests and status is modest; the hard part is throughput and fan-out, not bytes.

How the numbers affect the design

Core entities / data model

• NotificationRequest carries the producer's idempotency_key and the category (which channels and preferences are matched against, so it must be carried through); the system uses the key to reject a duplicate request (deep dive 2). It lives in a fast store (relational or key-value) keyed for dedup lookups.
• User holds the per-channel addresses: an email, a phone number, and a set of device tokens (one per installed app/device) used to push to that device. A user with three devices has three push addresses.
• Preference is the opt-out / channel-choice source of truth, queried during processing and again at the last safe point before sensitive provider sends.
• DeliveryAttempt is the status record per (request, user, channel) — append/update-mostly, used for status tracking and retries.

API design

The contract is producer-facing: other internal services call it to send notifications. There is no "fetch my notifications for the user" endpoint here beyond in-app retrieval; the system is push-oriented, driven by producers.

A key property: sending is accepted asynchronously. The API validates and enqueues the request, then returns immediately with 202 Accepted and an id — it does not wait for the message to actually reach the user's phone. Delivery happens in the background pipeline.

Send a notification

POST /api/notifications
Header: Idempotency-Key: <producer-generated-uuid>
Body: {
  "recipient_ids": ["user_123"],        // one or many user ids (or a segment id for broadcasts)
  "template_id": "price_drop",
  "channels": ["push", "email"],        // optional; default to the user's preferred channels
  "category": "marketing",
  "data": { "product": "Monitor X", "price": 249 }
}
202 Accepted
Returns: { "request_id": "...", "status": "accepted" }

Accept a request to notify one or more users. The server validates the payload, checks the Idempotency-Key for a duplicate, persists the request, and enqueues it — then returns 202 Accepted (the request is accepted for processing, not yet delivered). The Idempotency-Key header is what makes a retried POST safe: a network timeout that causes the producer to resend must not create a second set of notifications (deep dive 2).

Check delivery status

GET /api/notifications/{request_id}
200 OK
Returns: { "request_id": "...", "attempts": [ { "user_id": "...", "channel": "push", "status": "delivered" }, ... ] }

Let a producer poll the outcome of a request — per user, per channel. Because delivery is asynchronous, this is how a producer learns whether the message was delivered, bounced, or permanently failed.

Update preferences

PUT /api/users/{user_id}/preferences
Body: { "category": "marketing", "channel": "email", "enabled": false }
200 OK

Set a user's channel/category preference — this is the functional unsubscribe (deep dive 3). Once enabled is false, the pipeline must drop marketing email for this user before any provider call, including messages that were already waiting in a queue.

High-level architecture

Two flows make up the system: an ingestion path that accepts, dedups, and enqueues a request, and a delivery pipeline that turns a request into per-channel sends with retries. We'll walk each on its own, then combine them.

1. Ingestion path

The ingestion API does only fast, cheap work: validate, dedup on the idempotency key, persist, and enqueue. It never calls a provider.

flowchart LR
    Producer[Producer Service] -->|"POST /notifications + Idempotency-Key"| API[Ingestion API]
    API -->|"check idempotency key"| Dedup[(Idempotency Store)]
    API -->|"persist request"| ReqDB[(Request DB)]
    API -->|"enqueue"| InQ[[Ingestion Queue]]
    API -->|"202 Accepted"| Producer

1. A producer service sends a notification request with an Idempotency-Key.
2. The API does an atomic conditional insert of the key (a unique constraint on idempotency_key / INSERT ... ON CONFLICT / put-if-absent): if the insert wins, it's a new request; if it conflicts, the key was seen and the API returns the original request_id without enqueueing again — the duplicate is dropped here (deep dive 2).
3. For a new request, the API persists it and enqueues it on the ingestion queue.
4. The API immediately returns 202 Accepted — it does not wait for delivery.

Returning fast and pushing the real work onto a message queue is what lets the API absorb ~60K/sec bursts without each producer waiting on a slow provider.

2. Delivery pipeline

The pipeline takes a queued request, applies preferences and throttling, renders the template, fans out to per-channel queues, and lets per-channel workers call the external providers.

flowchart LR
    InQ[[Ingestion Queue]] --> Proc[Processor / Fan-out Worker]
    Pref[(Preferences + Throttle State)] --> Proc
    Tmpl[(Templates)] --> Proc
    Proc -->|"push messages"| PushQ[[Push Queue]]
    Proc -->|"email messages"| EmailQ[[Email Queue]]
    Proc -->|"SMS messages"| SMSQ[[SMS Queue]]
    PushQ --> PushW[Push Workers] --> PushP[(Push Providers - APNs/FCM)]
    EmailQ --> EmailW[Email Workers] --> EmailP[(Email Provider)]
    SMSQ --> SMSW[SMS Workers] --> SMSP[(SMS Provider)]
    PushW -->|"record status / retry"| Status[(DeliveryAttempt DB)]
    EmailW -->|"record status / retry"| Status
    SMSW -->|"record status / retry"| Status

1. The processor pulls a request from the ingestion queue. For a broadcast targeting many users, it expands the recipient list in batches (deep dive 2) rather than all at once.
2. For each recipient, it checks preferences and throttling: drop channels the user opted out of, skip if the user is over their rate limit or in quiet hours (deep dive 3).
3. It renders the template for each surviving channel, filling placeholders with the request's data and the user's locale.
4. It places one message per (user, channel) onto that channel's queue — push, email, or SMS each have their own queue and worker pool. The in-app channel (8% of volume) is the exception: rather than calling an external provider, it is a pull-based channel — the processor simply writes the notification to a per-user inbox store that the client fetches and marks read (see Extensions), which is why it does not appear in the push/email/SMS delivery diagrams.
5. Each channel's workers pull from their queue, re-check opt-out for channels where a stale send would be serious (especially email/SMS), and call the matching external provider — APNs/FCM for push, an email provider, an SMS provider.
6. Workers record the outcome in the DeliveryAttempt store and retry transient failures (deep dive 1).

Separate per-channel queues are the key structural choice: if the SMS provider slows down, only the SMS queue backs up — push and email keep flowing.

3. Combined architecture

Putting it together, with retries and a dead-letter queue for messages that exhaust their retries:

flowchart LR
    Producer[Producer Service] -->|"POST + Idempotency-Key"| API[Ingestion API]
    API -->|"dedup"| Dedup[(Idempotency Store)]
    API -->|"persist"| ReqDB[(Request DB)]
    API -->|"enqueue"| InQ[[Ingestion Queue]]
    InQ --> Proc[Processor / Fan-out Worker]
    Pref[(Preferences + Throttle State)] --> Proc
    Tmpl[(Templates)] --> Proc
    Proc -->|"per-channel messages"| PushQ[[Push Queue]]
    Proc --> EmailQ[[Email Queue]]
    Proc --> SMSQ[[SMS Queue]]
    PushQ --> PushW[Push Workers] --> PushP[(APNs/FCM)]
    EmailQ --> EmailW[Email Workers] --> EmailP[(Email Provider)]
    SMSQ --> SMSW[SMS Workers] --> SMSP[(SMS Provider)]
    PushW -->|"status"| Status[(DeliveryAttempt DB)]
    EmailW -->|"status"| Status
    SMSW -->|"status"| Status
    PushW -.->|"permanent failure"| DLQ[[Dead-Letter Queue]]
    EmailW -.->|"permanent failure"| DLQ
    SMSW -.->|"permanent failure"| DLQ

Queues at each boundary (ingestion, per-channel) keep the stages decoupled, so a burst at the front or a slow provider at the back does not stall the middle. The dotted edges to the dead-letter queue are where a message that keeps failing finally leaves the retry loop for inspection, instead of retrying forever and clogging the channel (deep dive 1).

Deep dives

1. Multi-channel delivery pipeline and reliability

The core of the system is turning an accepted request into a delivered message reliably, across channels, when the external providers are imperfect.

Queue-based decoupling. Every stage hands off through a queue. The ingestion API enqueues; the processor reads and enqueues to per-channel queues; channel workers read those. This means each stage runs at its own pace, scales independently, and survives the next stage being slow or down — the work waits safely in the queue instead of being lost or blocking the caller.

Per-channel worker pools and provider adapters. Each channel has its own queue and its own pool of workers, and each worker talks to its provider through a small adapter (a thin wrapper that translates your internal message into that provider's API). This isolates failures: if the SMS provider is throttling you, the SMS queue grows and SMS workers slow down, but push and email are untouched. It also lets you scale channels independently — push is 70% of volume, so it gets the most workers.

Retries with exponential backoff. Provider calls fail transiently (timeouts, 5xx, rate-limit responses). Retry, but back off exponentially — wait 1s, then 2s, 4s, 8s, with a little random jitter so a fleet of workers doesn't retry in lockstep and hammer a recovering provider.

Dead-letter queue for permanent failures. Some failures will never succeed: an invalid phone number, a 400 from the provider, a dead device token. After a bounded number of retries, move the message to a dead-letter queue instead of retrying forever. The dead-letter queue keeps "poison" messages from clogging the channel and gives you a place to inspect and alert on failures.

Device-token cleanup via provider feedback. A dead device token is not a DLQ problem — the provider tells you about it directly in the send response, and that permanent signal should automatically remove the token from the User store. FCM returns UNREGISTERED / 404 and APNs returns BadDeviceToken / Unregistered when a token is no longer valid (the app was uninstalled or the token rotated). On seeing one of those responses, the push worker deletes that device_token from the user's addresses so future sends skip it — rather than letting the same dead token keep failing and draining into the DLQ on every broadcast.

Distinguish transient from permanent. Retry transient errors (timeout, 429, 503); do not retry permanent ones (invalid address, hard bounce, opt-out) — send those straight to the dead-letter queue or mark them failed. Blindly retrying a permanent failure wastes work and can worsen your sender reputation.

A provider that is down for an extended period is a different problem. The dead-letter queue is for per-message permanent failures — a bad number, a hard bounce — not for a whole channel's provider being unavailable. If you blindly retry against a dead provider, every message eventually exhausts its retries and the entire channel's traffic drains into the dead-letter queue and is effectively lost. Instead use a circuit breaker: once a provider is failing consistently, stop hammering it and let the messages wait safely in the channel queue until it recovers, and/or fail over to a secondary provider for that channel. Reserve the dead-letter queue for messages that will never succeed, not for a provider-wide outage.

At-least-once, not exactly-once. Queues redeliver if a worker crashes after sending but before acknowledging, so a message can be processed more than once — at-least-once. You don't get exactly-once for free from a queue, and you cannot make an external provider call perfectly atomic with your database update. You reduce duplicates with idempotency keys, send-level dedup, and provider features such as push collapse keys or provider idempotency tokens where they exist (deep dive 2).

flowchart LR
    Producer[Producer] --> API[API]
    API -->|"call"| APNs[(APNs)]
    APNs --> EmailP[(Email Provider)]
    EmailP -->|"return 200"| Producer
    API -.->|"blocks here for<br/>the slow provider"| EmailP
    style API fill:#fee2e2,stroke:#dc2626,color:#7f1d1d

flowchart LR
    Producer[Producer] --> API[API]
    API -->|"enqueue"| Q[[Queue]]
    API -->|"202 (fast)"| Producer
    Q -.->|"background"| Proc[Processor]
    Proc --> ChanQ[[Per-channel Queues]]
    ChanQ --> Workers[Workers]
    Workers --> Providers[(Providers)]
    Workers -.->|"retry w/ backoff,<br/>then DLQ"| DLQ[[Dead-Letter Queue]]
    style API fill:#dcfce7,stroke:#16a34a,color:#14532d

2. Idempotency, dedup, and fan-out

Three related problems: a retried request must not produce duplicate notification work; the same notification should not reach a user twice in normal retries; and one event must fan out to millions of users without melting a provider.

Idempotency key on the request. The producer attaches a unique idempotency key to each send. A naive "check if the key exists, then insert it" is a check-then-write with a TOCTOU race: two concurrent copies of the same retry both see "not seen" and both enqueue. The dedup must instead be an atomic conditional insert — a unique constraint on idempotency_key with INSERT ... ON CONFLICT (or a put-if-absent) — so exactly one insert wins and the loser reads back the winner's request_id. Store request status alongside the key, not just the key, so a duplicate that arrives while the first request is still in_flight (or landed as failed) is handled correctly rather than treated as complete.

Producer sends "send receipt for order 555", key = abc-123
  API: INSERT key abc-123 (status=in_flight) succeeds → persist + enqueue → 202 {request_id: R1}
Producer times out, retries with the SAME key abc-123
  API: INSERT conflicts → read row → return {request_id: R1} (status in_flight/done), do NOT enqueue again
  → exactly one set of notifications, not two

Per-(user, channel) dedup deeper in the pipeline. At-least-once queues mean a worker might process the same message twice. Guard the actual send with a dedup key like (request_id, user_id, channel): reserve that key before calling the provider, and record the final outcome after the provider responds. If the same queue message is redelivered before any provider call happened, the worker sees the reservation and skips duplicate work.

Both stores need a TTL, not permanent retention. The producer idempotency key is kept for a bounded window (e.g. 24–48h, matching the longest the producer would still be retrying) — keep it forever and storage grows without bound; expire it too soon and a late retry re-sends. The per-send (request_id, user_id, channel) reservation needs a TTL long enough to outlive a provider call but short enough to self-heal: if a worker crashes mid-reservation, the key should expire so the message can be retried rather than being stuck reserved forever.

There is one important edge case: a worker can crash after the provider accepts the send but before your system records the final status. On retry, your database alone cannot prove whether the provider already sent it. Use provider-side idempotency tokens when the provider supports them, push collapse keys for replaceable mobile notifications, and stable message identifiers for in-app notifications. For email/SMS, where provider dedup is weaker, the honest guarantee is "best-effort duplicate suppression," not perfect exactly-once delivery.

Fan-out without melting a provider. A broadcast to 10M users is not one giant send. The 10M recipient list comes from an audience/segment query (e.g. "all users following team X"), and expanding it must itself be parallelized/sharded across many workers — one sequential pager that walks all 10M rows is the bottleneck, not the provider. Split the audience by user-id range (or by segment shard) and hand each shard to a different fan-out worker; each worker expands its slice in batches — pull a page of users, enqueue their per-channel messages, repeat — so the work streams onto the channel queues in parallel rather than materializing 10M messages at once or trickling them out from a single processor. The channel workers then drain those queues at a pace that respects each provider's rate limits (APNs/FCM and email/SMS providers all cap throughput). The queue is what converts a 10M-at-once spike into a steady, bounded stream.

The queue decouples arrival rate from delivery rate: the ~170K sends/sec figure is how fast messages are generated, but how fast they're delivered is bounded by each provider's throughput. So a 10M-user broadcast drains over minutes rather than seconds — which is fine for a non-urgent broadcast. Urgent traffic (a security code) should not wait behind it; route it through a separate priority lane so it isn't stuck behind a slow-draining bulk send.

flowchart LR
    Event[Broadcast event] --> Segment[(Segment query)]
    Segment -->|"10M target users"| Shard{"Shard by<br/>user-id range"}
    Shard --> WA["Worker A: shard [00–3F]<br/>pages in batches of ~1,000"]
    Shard --> WB["Worker B: shard [40–7F]<br/>pages in batches of ~1,000"]
    Shard --> WC["Worker C: ..."]
    WA -->|"enqueue push msgs"| PushQ[[Push Queue]]
    WB -->|"enqueue push msgs"| PushQ
    WC -->|"enqueue push msgs"| PushQ
    PushQ --> PushW["Push workers drain at<br/>provider-safe rate<br/>(APNs/FCM limits)"]
    style Shard fill:#e0f2fe,stroke:#0284c7,color:#0c4a6e

3. User preferences, throttling, and quiet hours

The system must never spam a user or send to someone who opted out. Check policy early in the processor to avoid unnecessary work, and check again at the channel worker before user-visible provider calls where stale preferences would be serious.

Preferences and opt-out. Each user has per-(category, channel) preferences: marketing email off, security SMS on, and so on. The processor reads these and drops any channel the user disabled before rendering or enqueuing. Channel workers should re-read the opt-out state immediately before email/SMS provider calls, because a user may unsubscribe while a queued message is waiting. A transactional/critical category (a login code, a security alert) may ignore marketing preferences, but a marketing message must always honor the opt-out — getting this wrong is both a bad experience and, for email/SMS, often a legal violation.

Functional unsubscribe. Opt-out has to actually work end to end: an email's unsubscribe link and an SMS "STOP" reply must flip the user's Preference to disabled, and the next send must see that and drop the channel. An unsubscribe that doesn't stop future mail erodes trust and harms your sender reputation.

Per-user throttling. Cap how many notifications a single user receives in a window (e.g. no more than N marketing pushes per hour). This is per-user rate limiting: keep a small per-user counter (in a fast store, with a TTL on the window) and skip or defer sends over the cap. It protects the user from a misbehaving producer or a coincidental pile-up of events.

Batching and digests. When many notifications target one user in a short window, coalesce them into a single digest — "You have 12 new likes" rather than 12 separate pushes. Digesting both improves the experience and cuts provider cost and load.

Quiet hours. Respect the user's local quiet hours (e.g. don't push between 10pm and 8am their time). For a non-urgent notification that lands in quiet hours, defer it to the end of the window rather than dropping it — schedule it to send when quiet hours end. But deferring every message in a timezone to the exact same instant (8:00am) creates a self-inflicted thundering herd — millions of sends fire simultaneously at the boundary. Add jitter: spread the deferred sends across a spatter of minutes as the window opens (e.g. each message releases at a random offset into the first N minutes after 8:00am) rather than releasing them all at the instant quiet hours end. Urgent/transactional messages (a security code) bypass quiet hours.

Processor, per recipient:
  1. opt-out?      → channel disabled for this category? drop the channel
  2. throttle?     → over per-user cap this hour? skip or defer
  3. quiet hours?  → in user's quiet window + non-urgent? defer to window end + jitter
  4. batch?        → many pending for this user? coalesce into a digest
  → only what survives gets rendered and enqueued to a channel

Channel worker, immediately before provider call:
  5. stale opt-out? → user unsubscribed while queued? mark skipped

Tradeoffs & bottlenecks

• Asynchronous vs synchronous delivery. Async (queues) decouples producers from slow providers and absorbs bursts, at the cost of eventual delivery and a status-polling API instead of an inline result. This is the central tradeoff, and async is the standard call.
• At-least-once vs exactly-once. True exactly-once across external providers is impractical; you get at-least-once from the queue and reduce duplicates with idempotency, dedup, and provider-specific keys. Accept rare visible duplicates on weaker channels rather than risk silently dropping important messages.
• Per-channel isolation vs more infrastructure. Separate queues and worker pools per channel isolate failures and scale independently, but mean more queues and workers to operate than one shared pipeline.
• Throttling/digesting vs immediacy. Batching and per-user caps protect users from spam but delay individual messages; urgent categories must opt out of batching and quiet hours.
• Provider dependency and rate limits. External providers (APNs/FCM, email, SMS) are the real bottleneck — they rate-limit you and have outages. Backoff, per-channel isolation, and a dead-letter queue contain the damage, but you can't deliver faster than the providers accept.
• Fan-out spikes. A broadcast to millions is a self-inflicted thundering herd; batched expansion plus queue-metered draining is what keeps it from overwhelming providers and your own workers.
• Status-store write load. Recording a DeliveryAttempt per (user, channel) at ~60K/sec is significant write volume; it's update-mostly and can be batched or written to a store tuned for high write throughput.

Extensions if asked

If the interviewer wants to push beyond the core delivery system, add only the extension that changes the design discussion:

• Scheduled and delayed notifications. Let producers schedule a send for a future time (or recurring); needs a scheduler/timer store that releases requests into the ingestion queue when due.
• Delivery analytics and bounce handling. Aggregate delivered/opened/bounced rates per template and channel; feed hard bounces and dead device tokens back into cleaning up addresses.
• Sender-reputation management. Warm up sending domains/IPs, monitor spam-complaint and bounce rates, and throttle to protect deliverability with email/SMS providers.
• Priority lanes. Separate high-priority/transactional traffic (login codes) from bulk marketing so a marketing fan-out can't delay a security code — e.g. dedicated high-priority queues and workers.
• In-app inbox. A pull-based store of in-app notifications the client fetches and marks read, separate from the push/email/SMS push channels.

What interviewers look for & common mistakes

What interviewers usually reward:

• Queue-based, asynchronous acceptance — 202 Accepted then background delivery, so producers are decoupled from slow providers.
• Per-channel queues and worker pools with provider adapters, so a failing channel is isolated and channels scale independently.
• At-least-once delivery made safer — an idempotency key on the request, a dedup key on the send, and provider idempotency/collapse keys where available, because queues redeliver and provider calls are external side effects.
• Retries with exponential backoff and a dead-letter queue, distinguishing transient from permanent failures.
• Batched fan-out for broadcasts, metered by the queue so providers aren't overwhelmed.
• Preferences, opt-out, per-user throttling, digests, and quiet hours enforced in the processor, with opt-out re-checked by channel workers for sensitive channels and a functional unsubscribe.

Before you finish, do a quick mistake check:

• Did you accept requests asynchronously instead of calling providers inline?
• Did you give each channel its own queue and worker pool so failures are isolated?
• Did you make sends idempotent where possible (request key + per-send dedup + provider keys where available) for at-least-once delivery?
• Did you retry with backoff and route permanent failures to a dead-letter queue?
• Did you expand large fan-outs in batches instead of all at once?
• Did you honor preferences/opt-out, re-check sensitive opt-outs before provider calls, and apply per-user throttling, digests, and quiet hours?