Design an A/B Testing System

Experimentation Platform System Design

Overview

Every product team eventually wants to answer the same question with numbers instead of opinions: does this change actually help? An A/B testing system — the kind that powers experimentation at Google, Meta, Netflix, and internal platforms like LaunchDarkly and Statsig — lets you ship a change to a random slice of users, hold the rest as a control, and measure the difference in a metric you care about (conversion, revenue, retention). You define an experiment with two or more variants, the system randomly assigns each user to one, delivers that assignment to the app fast, logs when the user was actually exposed, collects the outcome events, and runs a statistics pipeline that tells you whether the difference is real or noise.

It sounds like "flip a coin per user and count clicks," and the happy path really is that simple. It is labeled "Medium" because the parts that make it correct are subtle and commonly botched: assignment must be stable (a user must never flip variants mid-experiment) and stateless (you cannot store a row per user per experiment at billions of users); the statistics must respect the peeking problem and guardrails like sample ratio mismatch; and the exposure event has to fire at exactly the right moment or your analysis silently measures the wrong population.

A strong answer builds assignment as a deterministic hash so it needs no per-user storage, delivers config through an SDK with a cached snapshot so the app never blocks on a network call, logs exposure at the trigger point where the user is actually bucketed, and runs a statistics pipeline that reports confidence intervals and either fixes the analysis horizon up front or uses a valid sequential test — never naive p-value watching.

This design shares its assignment core with consistent hashing — both map an identifier through a hash into buckets — but the goals differ: consistent hashing minimizes key movement when nodes change, while A/B assignment wants a fixed, reproducible mapping from user to variant that is independent across experiments.

Out of scope (say so): the experiment-config UI, the feature-flag rollout tooling beyond the assignment layer, and the data-warehouse BI dashboards that visualize results — we cover the pipeline that produces the numbers, not the charting on top.

Functional requirements

• Define an experiment. Configure variants, allocation percentages (e.g. 50/50 or 90/10), a targeting rule (which users are eligible), and the metrics to measure. Support a gradual ramp (start at 1%, increase to 50%).
• Assign users to variants. Given a user (or device/session) and an experiment, return the variant. The assignment must be stable across requests and sessions and consistent across server and client.
• Deliver the assignment. The application must learn which variant a user is in with very low latency — the render path cannot wait on a slow service.
• Log exposure. Record an exposure event when a user is bucketed into a variant and could see the treatment — the anchor for all analysis.
• Collect outcomes. Ingest the metric events (clicks, purchases, session length) that experiments measure, and join them to exposures.
• Analyze and decide. Produce per-variant metrics with statistical significance, run data-quality guardrails, and support a ship / rollback decision.

Out of scope (state it): the config authoring UI, per-experiment access control, and downstream causal-inference research tooling. We assume outcome events already flow through the company's event pipeline.

Non-functional requirements

• Assignment is stable and stateless. A user always gets the same variant for a given experiment, computed on demand with no per-(user, experiment) storage. Reassigning a user mid-experiment corrupts the results.
• Very low assignment latency. Assignment sits on the request/render path. It must be sub-millisecond — effectively a hash — never a database round-trip per evaluation.
• Independent experiments. Two experiments running at once must assign independently — a user's variant in experiment A must not correlate with their variant in experiment B (unless deliberately made mutually exclusive).
• High write throughput for logging. Exposure and outcome events are high-volume, append-only, and analytics-bound. The event pipeline must absorb them without backpressure onto the app.
• Statistically correct analysis. The pipeline must report valid confidence intervals, detect SRM, and avoid the peeking trap. Correctness here is the product.
• Eventual consistency is fine for results. Dashboards that lag reality by minutes are acceptable; assignment correctness and event completeness are not negotiable.

Estimations

State assumptions; the goal is to justify the stateless-assignment and event-pipeline decisions, not to be exact.

Rounded assumptions

• Users: ~100M monthly active, ~10M concurrent at peak.
• Concurrent experiments: ~500 running across the product at any time.
• Assignment evaluations: every request may evaluate several experiments. At ~200K requests/sec peak × ~10 experiments each = ~2M assignment evaluations/sec. This must be pure CPU (a hash), not I/O.
• Assignment latency budget: < 1 ms, ideally microseconds — it runs inside the request path many times per request.
• Exposure events: a user hits an experiment on most sessions; ~10M concurrent users × ~1–2 exposures per active minute ≈ ~170K–350K exposure events/sec (a busier user base with more exposures pushes this toward ~1M/sec).
• Outcome events: clicks, views, purchases — the general event firehose, often millions/sec, of which experiments consume the relevant subset.
• Config size: 500 experiments × (variants + targeting rules) ≈ a few hundred KB of JSON — small enough to ship the whole config to every SDK and cache it.

How the numbers affect the design

Core entities / data model

Two kinds of data live here: configuration (small, relational, read-heavy — the experiment definitions) and events (huge, append-only, analytics-bound — exposures and outcomes). Assignment itself stores nothing — it is derived on demand.

The salt field is the linchpin of independent assignment: hashing user_id + salt (a distinct salt per experiment) means the same user's bucket in experiment A is uncorrelated with their bucket in experiment B. Reusing one global salt would make every experiment assign the same users to "position 0," so a user in treatment for A would systematically also be in treatment for B — a hidden confound.

Assignment produces no stored row. The ExposureEvent is not the assignment — it is the log that the assignment was acted on. This distinction matters: you can compute a user's variant a million times, but you only log exposure the one time they actually reach the code path that shows the treatment.

API design

Three surfaces: an SDK-facing assignment/config API, an event-logging API, and a config/results API for operators.

Fetch config snapshot (SDK bootstrap)

GET /api/v1/config?sdk_key=...&app_version=...
200 OK
Returns: {
  "version": 8412,
  "experiments": [
    {
      "experiment_id": "exp_checkout_button",
      "salt": "exp_checkout_button.v1",
      "status": "running",
      "targeting": { "country": ["US", "CA"] },
      "variants": [
        { "name": "control",   "allocation": 0.5, "config": { "color": "blue"  } },
        { "name": "treatment", "allocation": 0.5, "config": { "color": "green" } }
      ]
    }
  ]
}

The SDK fetches the full snapshot once at startup and then polls (or receives a push) for changes. Assignment happens locally in the SDK from this snapshot — no per-evaluation network call. version lets the SDK skip an unchanged config.

Assign (in-SDK, local — shown as a logical call)

assign(user_id, experiment_id) → variant_name        // pure function, no I/O

Deterministic and local. The SDK hashes user_id with the experiment's salt, checks targeting, and returns the variant. This is the microsecond hot path; it never leaves the process.

Log an exposure event

POST /api/v1/exposure
Body: {
  "user_id": "u_123",
  "experiment_id": "exp_checkout_button",
  "variant": "treatment",
  "timestamp": 1752105600,
  "context": { "app_version": "5.2.0", "platform": "ios" }
}
202 Accepted

Fire-and-forget, buffered client-side and flushed in batches. Returns 202 because ingestion is asynchronous. Emitted at the trigger point — see deep dive 3. Outcome events flow through the company's existing POST /events firehose, not a bespoke endpoint.

Experiment CRUD and results

POST /api/v1/experiments            // create/define variants, allocation, targeting
PATCH /api/v1/experiments/{id}      // ramp allocation, stop, or roll back
GET  /api/v1/experiments/{id}/results
200 OK
Returns: {
  "experiment_id": "exp_checkout_button",
  "srm_ok": true,
  "metrics": [
    {
      "name": "checkout_conversion",
      "control":   { "n": 240120, "mean": 0.104 },
      "treatment": { "n": 239880, "mean": 0.111 },
      "abs_lift": 0.007, "rel_lift": 0.067,
      "ci_95": [0.003, 0.011], "p_value": 0.002
    }
  ]
}

Results are computed by the batch/stream stats pipeline and served from a precomputed store; the endpoint is a read. Note the response leads with srm_ok — if the sample ratio check fails, every downstream number is suspect and the UI should refuse to show a verdict.

High-level architecture

Three planes carry the system: a config/assignment plane (definitions flow out to SDKs, which assign locally), a logging plane (exposures and outcomes flow into an event pipeline), and an analysis plane (events are joined and turned into statistics). We walk each, then combine.

1. Config and assignment plane

flowchart LR
    Op["Operator<br/>(defines experiment)"] --> CfgSvc["Config Service"]
    CfgSvc --> CfgDB[("Config store<br/>experiments + variants")]
    CfgSvc --> CDN["Config snapshot<br/>(CDN / edge cache)"]
    CDN -->|"poll or push"| SDK["Client SDK<br/>(cached snapshot)"]
    App["Application code"] -->|"assign(user, exp)"| SDK
    SDK -->|"hash user + salt<br/>local, sub-ms"| App

The operator defines experiments in the Config Service, which stores them relationally and publishes a compact snapshot to a CDN/edge cache. Every SDK holds the snapshot in memory and computes assignments locally by hashing — so 2M evaluations/sec cost nothing in network terms. A config change (ramp, stop) publishes a new snapshot version that SDKs pick up within their poll interval.

2. Logging plane

flowchart LR
    SDK["Client SDK"] -->|"exposure at trigger"| Ingest["Event Ingest API"]
    App["App / services"] -->|"outcome events"| Ingest
    Ingest --> Log[["Partitioned event log<br/>(Kafka, keyed by user_id)"]]
    Log --> Raw[("Raw event store<br/>data lake / warehouse")]
    Log --> Stream["Stream processor<br/>(near-real-time SRM + counts)"]

Exposure and outcome events flow into a durable, partitioned log. Keying by user_id co-locates a user's exposures and outcomes on the same partition, which makes the downstream join cheap. Events land in the warehouse for batch analysis and simultaneously feed a stream processor that computes near-real-time counts and the SRM guardrail early.

3. Analysis plane

flowchart LR
    Raw[("Raw events")] --> Join["Join exposures ⋈ outcomes<br/>per user, per experiment"]
    Join --> Agg["Per-variant aggregates<br/>n, sum, sum of squares"]
    Agg --> Stats["Stats engine<br/>CI, p-value, SRM"]
    Stats --> Store[("Results store")]
    Store --> Dash["Experiment dashboard"]
    Stats -.->|"SRM fail → flag, block verdict"| Dash

The analysis job joins exposures to outcomes (only users with an exposure count), aggregates per variant into the sufficient statistics needed for a t-test (n, sum, sum of squares — not raw rows), runs the significance test and SRM check, and writes results to a store the dashboard reads. An SRM failure short-circuits the verdict.

4. Combined architecture

flowchart LR
    Op["Operator"] --> CfgSvc["Config Service"]
    CfgSvc --> CDN["Config snapshot / CDN"]
    CDN --> SDK["Client SDK<br/>(local hash assignment)"]
    App["App"] --> SDK
    SDK -->|"exposure"| Ingest["Event Ingest"]
    App -->|"outcomes"| Ingest
    Ingest --> Log[["Partitioned event log"]]
    Log --> Raw[("Warehouse")]
    Log --> Stream["Stream SRM + counts"]
    Raw --> Analysis["Join + stats engine"]
    Analysis --> Results[("Results store")]
    Results --> Dash["Dashboard / decision"]

The three planes are decoupled: assignment never touches the logging or analysis path, so a slow warehouse never affects render latency, and an assignment spike never backpressures analytics. The only shared artifact is the config salt and experiment_id, which appear in both the assignment function and the exposure event so the two can be reconciled.

Deep dives

1. Assignment and bucketing — stable, stateless, independent

This is the heart of the system and the part interviewers probe hardest. The requirement is deceptively strict: the same user must get the same variant every single time, across servers and clients, with no stored state, and independently per experiment.

Deterministic hash bucketing. Assignment is a pure function:

h = hash(user_id + ":" + experiment.salt)      // fast non-crypto hash: MurmurHash3 / xxHash (good avalanche), take low bits
bucket = h % 10000                              // 10,000 fine-grained buckets, 0..9999
variant = variant whose cumulative allocation range contains `bucket`

With a 50/50 split, buckets 0–4999 → control, 5000–9999 → treatment. Because the hash is deterministic, the same user_id and salt always land in the same bucket — assignment is stable with zero storage. Because it is pure CPU, it runs in microseconds and satisfies the 2M-evals/sec budget. (Crypto hashes like MD5 were used historically for their uniformity, but a fast non-crypto hash fits the microsecond latency budget far better — you don't need cryptographic strength to split traffic.) This is bucketing, and it is the same primitive as feature flags, with the difference that experiments log exposure and analyze outcomes.

Using 10,000 buckets (rather than assigning variants directly from h % 2) decouples the bucket space from the variant count: you can ramp, add variants, or shift allocation by moving bucket ranges without rehashing users.

Unit of randomization. What you hash is a primary design decision, not a detail. A logged-in user_id gives one consistent experience across devices and sessions; a device/cookie ID is all you have for logged-out traffic but splits the same person across devices and resets on cookie clear; a session ID rebuckets the same person every session. The unit sets which population you can even test (logged-out visitors, multi-device users, whole B2B accounts) and what breaks: randomize at the device level but analyze at the user level and you get contamination and an inconsistent experience, because one person straddles both arms.

Independence via per-experiment salt. If every experiment hashed on user_id alone, a user at "hash position 0.3" would be in the low 30% of every experiment simultaneously — treatments would correlate and confound each other. Mixing a distinct salt per experiment re-randomizes the mapping per experiment, so a user's bucket in A is statistically independent of their bucket in B. Worked example: if experiments A and B share a salt, hash(user + same_salt) puts each user in the same bucket for both, so a user in treatment for A is disproportionately in treatment for B — the two treatment groups are correlated and each confounds the other. Per-experiment salt decorrelates them.

Gradual rollout / ramp. To ramp from 1% to 50% treatment safely, you widen the treatment bucket range over time. The critical rule: enrolled users must not flip. With control at [0, 4999] and treatment growing from the top down, treatment owns [9900, 9999] at 1% and expands downward toward 5000 — [9800, 9999] at 2%, … [5000, 9999] at 50%. Each step is a strict superset of the last, so everyone already in treatment stays in treatment and you only pull in new buckets. Ramping by shrinking control or reshuffling would flip existing users' variants — a stability violation that poisons the data.

Mutually-exclusive experiments (layers). Two experiments that touch the same surface (say, two different checkout-button redesigns) must not both run on the same user, or their effects entangle and neither is interpretable. The fix is layers: each layer has its own shared bucket space (hashed with the layer's own salt) and the experiments in it own disjoint ranges, so a user lands in at most one experiment per layer. Different layers hash independently via their own salts, so experiments across layers overlap freely. This is Google's "layers and domains" model — layers give you mutual exclusion where interactions are a real risk, while still letting hundreds of unrelated experiments run concurrently on the same users.

2. Metrics and statistical analysis — from events to an honest verdict

Getting a number is easy; getting a correct number is the job. This deep dive is where interviewers separate people who have run experiments from people who have only read about them.

From events to per-variant metrics. The join is: for each user with an exposure in experiment E, gather their outcome events after exposure, compute their per-user metric (did they convert? how much did they spend?), then aggregate per variant. Crucially you only aggregate the sufficient statistics — count n, sum, and sum of squares — which is all a t-test needs; you never ship raw per-user rows to the stats step.

Significance testing. For a metric like conversion rate, compare control mean p_c and treatment mean p_t. Compute the difference, its standard error, and a test statistic (a two-sample z-test / t-test; at experiment scale n is huge so they coincide). This yields:

• A p-value — the probability of seeing a difference this large if the treatment truly had no effect. Small p means "unlikely to be pure noise." It is not the probability that the treatment works, and a common interview mistake is to state it that way.
• A confidence interval for the effect size. Report the CI, not just the p-value — [+0.3%, +1.1%] conversion tells you both direction and magnitude, whereas "p = 0.002" hides whether the win is meaningful. A 95% CI that excludes zero is equivalent to p < 0.05.
• Statistical significance is declared when p crosses your pre-set threshold (α, typically 0.05) — but only at the analysis point you committed to (see peeking, below).

Sample Ratio Mismatch (SRM) — the first guardrail. Before trusting any result, check that the observed split matches the designed split. If you allocated 50/50 but observe 240,120 vs 239,880 that is fine; if you observe 250,000 vs 230,000 a chi-squared test on the counts flags SRM. SRM almost always means a bug — assignment skew, an exposure logged asymmetrically across variants, a variant crashing before it logs, or a filter dropping one arm — and it invalidates the experiment because the two groups are no longer comparable. A tiny relative imbalance is significant at large n, which is exactly why SRM is such a sensitive tripwire.

The peeking problem and why you can't just watch p-values. The single most damaging mistake in experimentation:

Guardrail metrics. Alongside the primary metric, always evaluate guardrail metrics (latency, crash rate, revenue) that the change must not harm. Guardrails are typically checked at a more liberal threshold than the primary metric — often one-sided, since you only care about the harmful direction — because their job is to catch regressions cheaply, not to prove an effect. A guardrail breach blocks the ship even when the primary metric wins: a conversion lift that tanks latency or spikes crashes should not go out.

3. Config delivery and exposure logging — fast assignment, correct anchoring

Assignment is worthless if the app can't get it quickly, and analysis is worthless if exposure is logged at the wrong moment. This deep dive covers both edges of the pipeline.

Config delivery — SDK with a cached snapshot. The app must never block on a network call to learn a variant. The pattern:

1. On startup the SDK fetches the full config snapshot (all ~500 experiments — only a few hundred KB) from a CDN/edge cache and holds it in memory.
2. Assignment then runs locally against that snapshot — hash, check targeting, read off the variant — in microseconds, offline-safe, no per-eval I/O.
3. The SDK keeps the snapshot fresh by polling for a new version on an interval (simple, cache-friendly) or via a push/streaming channel for near-instant propagation (faster kill-switches, more infra). Serving the snapshot from the edge/CDN keeps config-fetch latency low globally.

Because the config is tiny and assignment is a local hash, this scales to millions of evals/sec at near-zero cost — the expensive-looking part (per-user variant lookup) simply doesn't exist.

Exposure logging — fire at the trigger point. The exposure event defines your analysis population, so when you log it is a correctness decision, not a detail:

The correct rule: log the exposure exactly when the user is bucketed at the point the treatment could take effect — when the checkout code path runs and reads the variant, not at app launch. This is often called "trigger-based" or "lazy" exposure. It ensures the analyzed population is exactly the users for whom the variant mattered, which maximizes sensitivity and keeps the two arms comparable.

De-duplicate on the analysis side: a user triggering the same experiment many times is one exposed user (first-touch), so the join counts distinct exposed users, not exposure rows.

Joining exposures to outcomes at scale. Both streams flow through the partitioned event log keyed by user_id, so all of a user's exposures and outcomes co-locate on one partition. The analysis job:

1. Filters outcomes to those occurring after the user's first exposure (an outcome before exposure can't be an effect of the treatment).
2. Groups by (experiment_id, variant, user_id) and computes each user's metric.
3. Rolls up to per-variant sufficient statistics (n, sum, sum of squares) feeding the stats engine.

Keying by user makes this a partition-local join rather than a giant shuffle, which is what lets it scale to millions of events/sec. Outcome events can also arrive out-of-order or late (offline clients, retries), so the join needs a grace / late-arrival window before analysis finalizes — otherwise late outcomes are silently dropped and the metric is understated.

sequenceDiagram
    participant App
    participant SDK
    participant Log as Event log
    participant Stats
    App->>SDK: user reaches checkout, ask variant
    SDK->>SDK: hash user + salt, pick variant
    SDK->>App: return treatment
    SDK->>Log: emit exposure at this trigger
    App->>Log: later, emit purchase outcome
    Log->>Stats: join exposure then outcome per user
    Stats->>Stats: aggregate, test, SRM check

Tradeoffs & bottlenecks

• Client-side vs server-side assignment. Client-side (SDK in the app) removes a network hop and works offline, but ships the full config to devices and can be tampered with or delayed by slow snapshot propagation. Server-side keeps config private and updates instantly, but adds a hop and requires the server to be on every render path. Many platforms do both: server-side for sensitive backend experiments, client-side for UI. The assignment hash is identical either way, so a user gets the same variant regardless of where it's computed — which is essential for cross-tier consistency.
• Real-time vs batch metrics. A streaming pipeline gives near-real-time counts and early SRM detection (catch a broken experiment in minutes), but exact, reprocessable analysis usually comes from batch jobs over the warehouse. The pragmatic split: stream the guardrails and health checks, batch the authoritative verdict.
• Fixed-horizon vs sequential testing. Fixed-horizon is simple and slightly more powerful per sample but forces you to wait for the full sample. Sequential testing lets teams stop early and peek safely at the cost of complexity and a modest power penalty. Choose based on how badly teams want to move fast.
• Full-snapshot config vs per-experiment fetch. Shipping every experiment to every SDK is trivial at a few hundred KB but grows with experiment count; at tens of thousands of experiments you'd segment the config by app surface or fetch on demand.
• Number of buckets vs allocation granularity. 10,000 buckets give 0.01% allocation resolution and clean ramps; too few buckets make fine ramps and many-variant splits lumpy.

Extensions if asked

Add only the extension that changes the design discussion:

• CUPED (variance reduction). CUPED uses each user's pre-experiment behavior as a covariate to subtract predictable variance from the metric, tightening the CI and reaching significance with far less traffic. High-leverage for slow-moving metrics; requires a pre-period per user.
• Holdouts. Keep a small global holdout that never receives any treatment, to measure the cumulative long-term impact of everything shipped — catches the "many small wins that collectively regress a guardrail" problem.
• Feature-flag kill-switch. Because a variant is a flag, flipping allocation to 0% instantly disables a bad treatment via the config push path — decouple "kill the treatment" from "conclude the experiment."
• Multi-armed bandits. Instead of a fixed split, a multi-armed bandit dynamically routes more traffic to better-performing arms to minimize regret. Great for short-lived optimization (headline selection); worse for clean causal estimates, since shifting allocation biases the naive difference and complicates inference.
• Heterogeneous treatment effects / segmentation. Break results down by segment (new vs returning, platform, country) to find where the effect concentrates — but pre-register segments or correct for multiple comparisons, or you'll "find" a spurious segment win by testing enough slices.
• Novelty and primacy effects. A treatment can show an inflated novelty effect — engagement spikes just because the change is new, then decays — or the opposite primacy effect, where users resist an unfamiliar change and the initial dip reverses. Mitigate by running long enough for the effect to stabilize and by analyzing new vs returning users separately.

What interviewers look for & common mistakes

What interviewers usually reward:

• Stateless hash bucketing as the assignment core — deterministic, sub-millisecond, no per-user storage — and explaining why the stored-assignment alternative is the wrong (50-billion-row) design.
• Per-experiment salt for independence, and knowing that a shared salt correlates variants across experiments.
• Stable ramping — widening bucket ranges without flipping enrolled users — and layers for mutually-exclusive experiments.
• Statistical correctness — reporting confidence intervals, stating the p-value's real meaning, and naming the peeking problem with a valid fix (fixed-horizon or sequential).
• SRM as a first-class guardrail that gates every verdict.
• Exposure logged at the trigger point, once per user, only for the affected population — and why page-load logging dilutes the effect.
• Decoupled planes so assignment latency, event logging, and analysis never contend.

Before you finish, do a quick mistake check:

• Did you make assignment a deterministic, stateless hash rather than a stored per-user row?
• Did you use a per-experiment salt so experiments assign independently?
• Did you ramp by adding buckets only, never re-randomizing enrolled users?
• Did you state what a p-value and a confidence interval actually mean — and report the CI, not just significance?
• Did you call out the peeking problem and pick fixed-horizon or sequential testing?
• Did you make SRM a gate that blocks the verdict when it fails?
• Did you log exposure at the trigger point, once per user, for the affected population only?
• Did you address overlapping-experiment interaction with layers where it matters?