Design a Friend-Played Game Counter
Social Counting & Set Intersection System Design
Overview
Open a game's page on a Roblox-style platform and two little numbers appear almost instantly: "12 of your friends played this" and "48,302 playing right now." Both feel trivial — just show a count — but each hides a hard problem at scale. The first is a set intersection: for a given (user, game) you need the size of friends(user) ∩ players(game), where a user has hundreds of friends and a popular game has millions of players. The second is a near-real-time concurrency count that spikes into a hot key the moment a game goes viral.
It is labeled "Hard" because the obvious answer to the first signal — "loop over my friends and check each one" — costs O(friends) lookups on every game-page render, and a page shows many games at once, so the naive design multiplies out to an impossible read load. And the live count, kept in one counter, melts under a viral game's write storm. A strong answer reaches for fan-out on write for the friend signal (the news-feed pattern), a sharded live-session set whose members are counted into a cached total for the live signal (the likes-system hot-key pattern), and accepts that both counts can be a few seconds stale.
In this walkthrough you'll scope the system, size it, pick a data model, design the API, draw the read and write paths, and then go deep on the friend-set intersection, the live "playing now" count, and the graph/has-played stores plus the caching that ties it all together.
Out of scope (say so): matchmaking and the game servers themselves, the friend-request flow (we assume the friend graph exists), leaderboards, and anti-cheat. We keep only the two counters and the queries that feed them.
Functional requirements
- Friends-who-played count. For a
(user, game)pair, show how many of the user's friends have ever played that game (e.g. "12 friends played"). Rendered on every game page. - Live player count. For a game, show how many people are playing right now in near-real-time (e.g. "48,302 playing").
- Batched for a page of games. A browse page shows many games; the client must fetch both signals for a whole page of games in one request, not one call per game.
- Record a play. When a user starts playing a game, record the has-played fact so the friend counter can update.
- Heartbeat / session lifecycle. A live session is kept alive by periodic heartbeats and ends on stop or when its heartbeat expires.
Out of scope (state it): the game runtime, matchmaking, friend-request management, and recommendation ranking. This system owns the friend-played count, the live count, and the play/heartbeat events that drive them.
Non-functional requirements
- Read-heavy. Both counts are read on every game-page render — far more often than a play or heartbeat is written. The read path must be cheap, cached, and batched.
- Near-real-time, not real-time. A live count that is a few seconds stale is fine — nobody notices "48,302" vs "48,310" for a moment. Same for the friend count. Eventual consistency is the norm.
- Handles hot keys. A viral game draws millions of concurrent players and huge play volume onto one
game_id. No single game's popularity may create a contended key that serializes its writes. - Approximate is acceptable. "50+ friends played" or a live count rounded to the nearest hundred is fine and much cheaper than an exact figure. This tolerance is what unlocks the cheap designs.
- Availability over strong consistency. A failed count read should degrade to a slightly stale (or hidden) number, not an error on the game page.
Estimations
State assumptions; the goal is to justify the data model and the counter designs, not to be exact.
Rounded assumptions
- Users: ~10M monthly, with a large daily-active slice.
- Games: ~100K.
- Friends per user: ~hundreds (say ~300 average), with a heavy tail — some users have thousands.
- Game-page renders/sec. Browsing and game pages dominate. Assume ~500K game-page renders/sec at peak, each needing both signals for the ~30 games on screen — so ~15M game-signal lookups/sec (batched into far fewer requests, but that many logical count reads).
- Plays/sec. Users start games far less often than they view pages: order of ~10K–50K play-starts/sec.
- Live sessions. Millions of concurrent sessions across all games, sending a heartbeat every ~15–30s — so hundreds of thousands of heartbeats/sec in aggregate.
- has-played rows. 10M users × the games each has played (tens to hundreds each) = hundreds of millions to low billions of
(user, game)facts.
How the numbers affect the design
| Signal from the numbers | Design decision |
|---|---|
| Reads (both counts) ≫ writes | Serve both counts from a cache, never by intersecting sets or scanning per read. |
| A page renders many games at once | Batch both signals for a whole page in one request. |
| O(friends) lookups per game × many games per render | Naive intersect-on-read is impossible; precompute or cache the friend count (deep dive 1). |
| One viral game takes millions of concurrent players | The live-session set is a hot key — shard it and count its fresh members into a cached total (deep dive 2). |
| Hundreds of millions of has-played facts | Store in a sharded wide-column / KV store partitioned to match the lookup (deep dive 3). |
| Counts may be a few seconds stale, and "50+" is fine | Async, eventually-consistent, optionally approximate counting is acceptable. |
Caution
Do not plan to compute the friend count by looping over a user's friends and checking each one on every render. With ~15M game-signal lookups/sec and many games per page, O(friends) lookups per game is impossibly expensive — the count must be precomputed or cached, not computed live per render.
Core entities / data model
Four things are stored: the friend edge (the social graph), the has-played fact (source of truth for who played what), the live session (source of truth for who is playing now), and the two derived counters.
| Entity | Key fields |
|---|---|
Friend (edge) |
(user_id, friend_id) — bidirectional; stored both directions, plus created_at |
HasPlayed (fact) |
(user_id, game_id) — composite key, plus first_played_at |
LiveSession |
(session_id) → user_id, game_id, last_heartbeat_at (expires on TTL) |
FriendPlayedCount |
(user_id, game_id) → integer (derived: how many of my friends played this) |
LivePlayerCount |
game_id → integer (derived: count of live session keys, cached; sharded set) |
The Friend edge is the social graph. Friendship is bidirectional, so the edge is stored both directions (a row for (a, b) and one for (b, a)), letting "give me a user's friends" be a cheap single-partition read (deep dive 3) — the same follow-graph pattern a news feed uses.
The HasPlayed fact is the source of truth for the friend signal: a user played a game or they didn't. The FriendPlayedCount is derived from it — a maintained number, never the source of truth. If it drifts or is lost, it can be rebuilt from the has-played facts and the friend graph.
The LiveSession is the source of truth for the live signal: a session is "live" until its heartbeat TTL expires. LivePlayerCount is a derived total obtained by counting the live session keys (those with a fresh heartbeat), cached and refreshed every second or two — never an incr/decr accumulator (deep dive 2).
Caution
Do not store the counts as the source of truth and treat the facts as optional. The has-played facts and live sessions are authoritative; the counts are derived caches of them. Keep only a count and you cannot rebuild after a bad increment, and you cannot recompute a user's friend count when a new friendship forms.
Where each lives. The friend graph and has-played facts live in a sharded wide-column / KV store (Cassandra-style) partitioned by user_id for cheap per-user reads. Live sessions and both counters live in an in-memory store (Redis or similar) that supports atomic increment, TTL keys, and set/bitmap operations.
API design
Five operations carry the system: the two batched reads, and the three writes that keep the counts fresh.
Get friends-who-played count
GET /api/games/{game_id}/friends-played (user comes from the auth token)
200 OK
Returns: { "game_id": "...", "friends_played": 12 }
Return how many of the caller's friends have played this game. Served from the maintained/cached FriendPlayedCount for the (user, game) pair — never by intersecting the friend set with the player set on the fly (deep dive 1). The value may be approximate ("50+").
Get live player count
GET /api/games/{game_id}/live-count
200 OK
Returns: { "game_id": "...", "live_players": 48302 }
Return how many people are playing right now. Served from a cached total that a background job maintains by counting the game's live session keys (across shards), a few seconds stale by design (deep dive 2).
Batched — both signals for a page of games
POST /api/games/signals
Body: { "game_ids": ["g1", "g2", "g3", ...] }
200 OK
Returns: { "g1": { "friends_played": 12, "live_players": 48302 },
"g2": { "friends_played": 0, "live_players": 130 }, ... }
The workhorse for a browse page. One request resolves both counts for every game on screen, so a page of 30 games is one round trip instead of 60 calls. The server fans the lookup across the cached friend counts and live counters in parallel.
Record a play
POST /api/plays
Body: { "game_id": "..." } (user from auth token)
202 Accepted
Returns: { "recorded": true }
Record that the caller played this game. Does an atomic conditional insert (INSERT ... IF NOT EXISTS) of the (user_id, game_id) has-played fact, and — only if that insert reports the fact was newly created — triggers the fan-out that bumps this user's friends' friend-played counters for that game (deep dive 1). The atomic insert's boolean is the only safe dedup signal; a separate read-then-write check would race and double-fan-out. Returns 202 because the fan-out is asynchronous.
Heartbeat / session lifecycle
POST /api/sessions/heartbeat
Body: { "session_id": "...", "game_id": "..." }
204 No Content
Start or refresh a live session. Each heartbeat upserts session_id → last_heartbeat into the game's live-session set and refreshes the key's TTL — the same operation for the first heartbeat and every later one; the live count is derived by counting fresh members, not bumped here. A session with no heartbeat within the TTL window is simply no longer counted (deep dive 2). An explicit DELETE /api/sessions/{session_id} drops it immediately, but correctness must not depend on that arriving.
High-level architecture
Two read paths serve the game page, and two write paths keep the derived counts fresh: a play event (feeds the friend count) and a heartbeat (feeds the live count). We'll walk each, then combine them.
1. Read path (both signals, batched)
The game page reads both counts from caches — never by intersecting sets or scanning sessions.
flowchart LR
Client["Client<br/>(game page)"] -->|"POST signals<br/>(page of game_ids)"| ReadSvc["Signals Service"]
ReadSvc -->|"per (user, game):<br/>read cached friend count"| FCache[("Friend-played counts<br/>(cached, per user+game)")]
ReadSvc -->|"per game:<br/>read cached live total"| LCounter[("Live count<br/>(cached total from<br/>live-session count)")]
ReadSvc -->|"both counts per game"| Client
- The client sends one batched request with the game ids on screen.
- For each game, the Signals Service reads the cached
FriendPlayedCountfor this(user, game)— an O(1) cache hit, computed on miss (deep dive 1). - For each game, it reads the game's live count from the cached total that a background job maintains by counting live session keys (deep dive 2).
- It returns both numbers per game in a single response.
Tip
Both signals fire on every render, so both must be O(1) cache reads. Never answer either by iterating a friend list or scanning live sessions at read time.
2. Write path A — a play event (feeds the friend count)
Recording a play writes the has-played fact once and, only on a genuinely new fact, fans out to the player's friends.
flowchart LR
Client["Client"] -->|"POST play"| PlaySvc["Play Service"]
PlaySvc -->|"insert-if-absent (user, game)"| Played[("HasPlayed store<br/>(sharded by user_id)")]
Played -->|"was it new?"| PlaySvc
PlaySvc -.->|"if new: enqueue fan-out"| Q[["Fan-out queue"]]
Q --> W["Fan-out workers"]
Graph[("Friend graph")] --> W
W -->|"for each friend f:<br/>increment friendPlayed(f, game)"| FCounter[("Friend-played counts")]
- The Play Service does an atomic conditional insert of the
(user, game)has-played fact —INSERT ... IF NOT EXISTS(a put-if-absent /SETNX), one round trip that both writes and tells you whether it wrote. - The store returns a boolean "was this new?" This boolean is the sole idempotency guard that gates the fan-out. A read-then-write check ("does the fact exist? no → insert") is a race: two concurrent replays of the same play both read absent and both fan out, double-counting. Only the atomic insert's own result is safe to trust.
- Only if new, it enqueues a fan-out job (off the request path).
- Workers look up the player's friends in the graph.
- For each friend
f, they incrementfriendPlayed(f, game). Because the fan-out queue is at-least-once (a worker crash re-delivers the job), the increment must itself be idempotent — dedup by(player, friend, game)(or makefriendPlayeda set of players you can re-add safely), so a redelivered job doesn't double-count. So when friendflater opens that game's page, their count is already computed. This is fan-out on write, exactly like a news feed (deep dive 1).
3. Write path B — a heartbeat (feeds the live count)
Every heartbeat records the session as live in the game's live-session set; the count is derived later by counting the still-fresh members, so nothing has to run on expiry.
flowchart LR
Client["Client"] -->|"POST heartbeat<br/>(session_id, game_id)"| HbSvc["Session Service"]
HbSvc -->|"upsert session_id<br/>→ last_heartbeat (+TTL)"| Sess[("Live-session set<br/>(sharded, per game)")]
Sess -.->|"stale members fall<br/>outside TTL window"| Drop["(no callback needed)"]
Recount["Recompute job:<br/>count members with<br/>score > now − TTL"] --> Sess
Recount -->|"sum → cached total"| LCounter[("Live count<br/>(cached, ~1-2s)")]
- A client sends a heartbeat for its
session_idandgame_id. - The Session Service upserts
session_id → last_heartbeatinto the game's (sharded) live-session set and refreshes the key's TTL — the same operation whether the session is new or existing. - A background recompute job counts, per game, the members whose last heartbeat is inside the TTL window (
score > now − TTL), sums the shards, and caches the total (~1–2s fresh). - A session that stops heartbeating (the player left, crashed, or lost network) simply falls outside the window and is no longer counted on the next recompute — no decrement, no expiry callback. Correctness relies on TTL freshness, not on an explicit stop arriving (deep dive 2).
4. Combined architecture
Putting them together: plays feed the friend count via fan-out; heartbeats keep the live-session set fresh and a background job derives the live count by counting it; the game page reads both from caches.
flowchart LR
Client["Client"] --> GW["API Gateway"]
GW -->|"POST play"| PlaySvc["Play Service"]
PlaySvc -->|"insert-if-absent"| Played[("HasPlayed store")]
PlaySvc -.->|"if new"| Q[["Fan-out queue"]]
Q --> W["Fan-out workers"]
Graph[("Friend graph")] --> W
W -->|"incr friendPlayed(friend, game)"| FCounter[("Friend-played counts")]
GW -->|"POST heartbeat"| HbSvc["Session Service"]
HbSvc -->|"upsert session→last_hb"| Sess[("Live-session set (sharded, TTL)")]
Sess -->|"count members<br/>score > now − TTL"| LCounter[("Live count (cached total)")]
GW -->|"POST signals (batched)"| ReadSvc["Signals Service"]
ReadSvc -->|"read friend count"| FCounter
ReadSvc -->|"read cached total"| LCounter
The has-played facts and live sessions are the source of truth; both counters are derived and rebuildable. Reads and writes are decoupled: the batched read never blocks on fan-out or heartbeat processing, so a viral game's write storm doesn't slow anyone reading its counts.
Deep dives
1. Friends-who-played — the set intersection at scale (the headline)
For a (user, game) you need |friends(user) ∩ players(game)|. A user has hundreds of friends; a popular game has millions of players. This intersection is rendered on every game page, and a page shows many games. The whole design turns on computing it cheaply. Here are three approaches, worst to best.
Intersect on read — scan the friend list, check has-played per friend (avoid — O(friends) per game, per render)
On each game-page render, loop over the user's friends and do a has-played point lookup for (friend, game) for each one, counting the hits. This is the direct translation of "how many friends played this."
Worked example — a user with 300 friends opens a browse page of 30 games:
per game: 300 has-played lookups (one per friend)
per page: 30 games × 300 = 9,000 lookups for ONE page render
at scale: ~500K renders/sec × 9,000 = ~4.5B lookups/sec ← impossible
The cost is O(friends × games-on-page) on every render, and renders are the highest-volume event in the system. It also scales the wrong way: your most-connected users (thousands of friends) are the most expensive to serve.
- Pro: no precomputation, no extra storage; always exactly correct at read time.
- Con: O(friends) lookups per game × many games per render = a read explosion that no cache of the raw facts can rescue, because the intersection itself is recomputed every time.
- Verdict: avoid — correct but hopelessly expensive on the hot read path. This is the naive answer the question is testing whether you avoid.
Precompute every (user, game) pair up front (works, with caveats — storage & write explosion)
Maintain a friendPlayed(user, game) count for every pair, always kept current. Then a read is a single O(1) lookup. The catch is keeping it current for all pairs.
Worked example — the size of the space:
pairs: 10M users × 100K games = 1 trillion (user, game) cells
— most are zero (you have no friend who played that game),
but you must maintain the non-zero ones as facts change
write fan-out on one play: when user U plays game G, EVERY friend of U
needs friendPlayed(friend, G) bumped — and if you also
eagerly materialize zeros, the space is astronomical.
Storing a dense table over all pairs is wasteful (most cells are zero), and the write cost of keeping it current is a fan-out on every play. The insight that rescues this is: you don't need all pairs, only the pairs that are non-zero — which is exactly what fan-out on write produces below. So "precompute everything" is really "precompute the pairs that matter, lazily," which is the next option.
- Pro: reads are a single O(1) lookup — the cheapest possible read.
- Con: materializing all pairs is storage you can't afford and mostly zeros; keeping it current still requires per-play fan-out. Naive full precomputation is wasted work for pairs no one ever views.
- Verdict: works only if you precompute sparsely — just the non-zero pairs, produced by fan-out on write. Full dense precomputation is a trap; the sparse version is the recommended design below.
Fan-out on write for the signal, with a cache-on-read fallback (recommended — the default at scale)
Turn the read-time intersection into a write-time increment. When a user plays a game, for each of their friends increment that friend's friendPlayed(friend, game) counter. The counter only ever holds non-zero, actually-relevant pairs, and a read is one O(1) lookup — the news-feed fan-out-on-write pattern applied to a counter.
Worked example — user U (300 friends) plays game G for the first time:
on play: for each of U's 300 friends f: friendPlayed(f, G) += 1
(300 small increments, async, off the request path)
on render: friend f opens G's page → read friendPlayed(f, G) → "12" (one lookup)
For the pairs never touched by any play, no cell exists and the count is simply 0 — so storage tracks only the non-zero intersections, not the trillion-cell space.
The hybrid fallback — cache-on-read. For cold or rare pairs you don't want to fan out to (or to self-heal drift), compute the intersection once on a read miss and cache it with a short TTL. To make that on-read compute fast, store each game's players as a roaring bitmap of player ids, keyed on a global user_id → dense-index mapping shared by every bitmap so a bitwise AND lines up. But for a small friend set (~hundreds), don't materialize a 10M-bit friend bitmap just to AND it — instead test each friend id for membership in the game's player bitmap (~hundreds of O(1) contains checks). The cost of the intersection is bounded by the smaller of the two sets, which is almost always the friend set. Whichever you use, the read is served from the cached number, and the compute happens at most once per TTL.
flowchart LR
Play["User plays game G"] -->|"for each friend f"| FO["increment<br/>friendPlayed(f, G)"]
FO --> Store[("Friend-played counts<br/>(non-zero pairs only)")]
Render["Friend f opens G page"] -->|"O(1) read"| Store
Store -.->|"cold pair / miss"| Compute["intersect friend-set ∩<br/>game player bitmap, cache w/ TTL"]
classDef good fill:#dcfce7,stroke:#16a34a,color:#14532d;
class Store good
The write volume this creates. Quantify the write side the way we quantified reads: ~10K–50K plays/sec × ~300 friends each ≈ several million counter increments/sec at steady state. That is the price of pushing the signal on write, and it is why the fan-out must be async, off the request path, and capped. Above a friend-count threshold (say ~5K friends) skip the push entirely and fall back to compute-on-read for that user's tail — otherwise one hot user's single play alone dumps thousands of increments into the queue.
The celebrity / hot problem — same as news feed, two directions:
- A hugely popular game: its player bitmap has millions of ids. Intersecting a friend set against it is still cheap (the friend set is small, ~hundreds), so the bitmap approach handles this fine — you test each friend for membership in the big bitmap, cost bounded by the small set.
- A hot user (someone with tens of thousands of friends): one play triggers tens of thousands of increment fan-outs. Cap the fan-out at the ~5K-friend threshold and fall back to compute-on-read for the tail, exactly as a news feed pulls celebrity posts at read time instead of pushing them.
Approximate is fine. The product only needs "12 friends" or even "50+ friends played." An approximate count is far cheaper — you can cap the counter, stop at "50+", and never need exactness. Nobody notices if it reads 11 instead of 12 for a few seconds.
- Pro: reads are O(1); storage holds only non-zero pairs; the intersection is paid once at write time, not on every render. The bitmap fallback makes on-read compute a bitwise AND, not a lookup loop.
- Con: fan-out cost on each play (bounded by friend count); a hot user needs the compute-on-read fallback; the count is eventually consistent. The increments must be idempotent (the queue is at-least-once), and any residual drift is corrected by a periodic recompute from
friends ∩ has-played. - Verdict: the default at scale. Fan out the signal on write for normal users, fall back to a TTL-cached intersection (over player bitmaps) for cold pairs and hot users, and accept an approximate, slightly-stale count.
The fan-out-on-write vs on-read tradeoff, explicitly. This is the same axis as a news feed. Fan-out on write (push a friendPlayed increment to each friend when you play) makes reads O(1) but pays per play and struggles with hot users. Fan-out on read (intersect friend set ∩ player set when the page loads) makes writes free but pays O(friends) per render — untenable here because renders vastly outnumber plays. The production answer is the hybrid: push for normal users, compute-and-cache on read for the hot/cold tail.
Warning
Do not try to precompute a dense table over all 10M × 100K pairs. Fan out on write so you store only the non-zero pairs, and fall back to a TTL-cached intersection for the rest. Reach for approximate ("50+") to make it cheaper still.
2. Live "playing right now" count — real-time concurrency per game
The live count is the number of active sessions on a game at this moment. Players send periodic heartbeats; a session is live until its heartbeat TTL expires. Two problems: keeping the per-game count correct, and surviving a viral game.
Counting: derive the count from the live sessions, don't accumulate it. The tempting design — increment a counter on start, decrement on end — quietly breaks, because you cannot reliably run a decrement at end-of-session. A session ends when its heartbeat TTL expires, and passive key-expiry fires no application callback — nothing runs your decrement. Worse, if start incremented a random shard, you have no record of which shard to decrement anyway. So the live count is derived by counting live session keys, not maintained by an incr/decr accumulator.
The concrete structure: keep a per-game sorted set of session_id → last_heartbeat (a heartbeat updates the member's score; each session's key also carries a TTL). The live count at any instant is the number of members whose score is greater than now − TTL — i.e. sessions that heartbeat recently enough to still be alive. A background job counts those live members per game and refreshes a cached total every ~1–2s; virtually every read hits that cached number. A few-seconds-stale count is acceptable.
A viral game is still a hot key — millions of sessions churning one game_id. Two ways to keep the count cheap under that load, both compatible with "count the live sessions":
- Shard the live-session set into N per-game sub-structures; a session lands on a random shard, the recompute counts each shard's live members and sums them. No single key takes more than 1/N of the churn. (Order N in the tens to low hundreds.)
- If you keep a sharded counter as the cached total, treat it explicitly as a cache periodically recomputed by counting live sessions (or maintained by an active reaper that scans for expired members, or by keyspace-notification events — both lossy under load). It is not maintained by decrement-on-passive-expiry. With random-shard increments a shard can even go individually negative; only the sum across shards is meaningful.
flowchart LR
S["session heartbeats<br/>on game G"] -->|"random shard"| Sh0["shard 0<br/>(session→last_hb)"]
S --> Sh1["shard 1"]
S --> Shn["… shard N-1"]
Sh0 --> Sum["COUNT members with<br/>score > now − TTL, per shard"]
Sh1 --> Sum
Shn --> Sum
Sum --> Total["sum → cached total<br/>(refreshed every ~1-2s)"]
classDef good fill:#dcfce7,stroke:#16a34a,color:#14532d;
class Sh0,Sh1,Shn good
Expiry correctness — rely on the TTL, not the stop event. The critical subtlety: you cannot trust an explicit "stop" to arrive. Players crash, close the tab, or lose network — the "stop" never fires. If the count depended on an explicit stop firing a decrement, it would drift upward forever. Instead, a session counts as live only while its last heartbeat is inside the TTL window; once it falls outside, it simply stops being counted on the next recompute — no callback required. An explicit stop is an optimization (drop the member immediately), never the mechanism.
Caution
Do not model the live count as an accumulator decremented on session end. Passive TTL expiry fires no callback to run a decrement, and a random-shard increment leaves nothing to decrement. Derive the count by counting live session keys (score > now − TTL) and cache the total; a session is live only while its heartbeat is fresh.
Distinct players vs concurrent sessions. One player on two devices is two sessions but one person. Decide which the product wants:
- Concurrent sessions ("48,302 playing") — count sessions; the derived counter above does this directly.
- Distinct concurrent players — deduplicate by
user_id. This needs a windowed / rebuilt set of live user ids per game (or per-shard HyperLogLog rebuilt each window and merged on read for an approximate figure) — you cannot use a plain, ever-accumulating HyperLogLog here because it is add-only and cannot shrink as sessions end; only the per-window rebuild lets the count fall as players leave. Note the tradeoff: a per-game exact live-user set reintroduces the hot-key / large-structure problem the sharded session counter avoided (and sharded session counters don't dedup across shards, so they can't answer this directly).
"Unique players today" — approximate distinct count. HyperLogLog fits this shape because a daily-unique figure is cumulative — it only ever grows, and HLL is add-only (no removal). For a viral game an exact set of tens of millions of ids is expensive to hold and merge, so use an approximate distinct count (HyperLogLog) — it estimates distinct players in a few KB with ~1-2% error instead of storing every id. This is a cumulative count, not the shrinking live-concurrent set above; don't reach for HLL when the set has to shrink. The product tolerance ("approximate is fine") makes this the right call for the cumulative case.
Warning
A viral game's live-session set is a hot key just like a viral post's like counter. Shard it, count fresh members across shards, cache the total, and accept a few-seconds-stale number. Don't reach for one atomic incr/decr counter — atomicity fixes lost updates, not contention, and there's no callback to decrement on passive TTL expiry anyway.
3. The social graph, the has-played store, and caching the hot answers
The two counters sit on top of two source-of-truth stores — the friend graph and the has-played facts — and a caching layer that makes both signals O(1) on the render path.
The social graph. Friendship is bidirectional, so store the edge both directions: a row for (a, b) and one for (b, a). Partition by user_id so "give me a user's friends" is a single-partition read — you need it cheap because fan-out (deep dive 1) reads a player's friend list on every new play. This is the same follow-graph-stored-both-ways pattern a news feed uses; the difference is symmetry (a friendship is mutual, a follow is directed), which is why both rows always exist as a pair.
New-friendship backfill. Fan-out-on-write only bumps counters at play time, so a friendship formed after a friend already played leaves the count wrong until something fixes it. Forming a new edge (A, B) must therefore enqueue its own backfill: for each game B has already played, friendPlayed(A, game) += 1, and symmetrically for each game A has played, friendPlayed(B, game) += 1. (These reuse the same idempotent, dedup-by-(player, friend, game) increments as the play-time fan-out.) For a pair whose counts are stale you can instead recompute-on-read — intersect the two users' played-game sets lazily on the next miss. Either way, a new friendship is a first-class event that mutates the derived counts, not just a play.
The has-played store. This is large — 10M users × the games each played = hundreds of millions to low billions of (user, game) facts. Store it in a sharded wide-column / KV store as point lookups, partitioned by user_id so a user's played-games are co-located. Two shapes work: a per-(user, game) key (existence check), or a per-user set of played game ids (a compact set you can read whole, useful for computing intersections in the fallback path). Partitioning by user_id makes "did this user play?" and "what did this user play?" cheap; the reverse question ("who played this game?") is served by the game's player bitmap (deep dive 1), not by scanning this store.
The player bitmap is a second write, and not free to keep in RAM. The game-partitioned player bitmap is a secondary index that inverts the user_id-partitioned has-played fact into a game_id-partitioned one — so a first play is two writes, the has-played fact plus a bit set in the game's bitmap, not one. And a big game's bitmap is single-digit to tens of MB, so you cannot hold all ~100K games' bitmaps in memory at once. Keep bitmaps resident only for hot / large games; for the long tail, page the bitmap in from the durable wide-column store on demand (and drop it under memory pressure).
Caching the hot answers. Both signals are read on every render, so both are cached with short TTLs, and eventual consistency is fine — nobody notices a few-seconds-stale "12 friends played" or "48,302 playing."
- The
friendPlayed(user, game)count is cached per pair; on a miss it is computed (intersection over the player bitmap) and backfilled (cache-aside). The miss can be expensive when the game's bitmap is cold — loading or building a multi-MB structure — so a burst of misses on one hot game must single-flight behind a per-key lock (one recompute, the rest wait and read its result), not fire N concurrent rebuilds of the same bitmap. - The per-game live count is cached as a summed total, refreshed every second or two rather than re-summed per read.
Batching a page of games. The browse page batches both signals for all its games in one request (the POST /api/games/signals endpoint). The service resolves the cached counts in parallel — one round trip for 30 games, not 60 calls.
The counts are derived, rebuildable caches. Neither counter is ever the source of truth. The friend count rebuilds from the has-played facts ∩ friend graph; the live count is inherently ephemeral (rebuild by scanning live sessions). If a count is lost or drifts, a background reconciliation recomputes it — off the hot path, because the read path always serves the maintained/cached number.
Warning
Keep the counts as derived caches, never the source of truth. A new friendship, a corrected has-played fact, or a lost counter must all be recoverable by recomputing from the graph and facts — which is impossible if the count is all you kept.
Tradeoffs & bottlenecks
- Fan-out on write vs on read (friend count). Push a friendPlayed increment per friend on each play (cheap O(1) reads, per-play write cost, hot-user trouble) vs intersect on read (free writes, O(friends) per render — untenable). Hybrid: push for normal users, cache-on-read for the hot/cold tail.
- Dense vs sparse precomputation. A dense table over all 10M × 100K pairs is unaffordable and mostly zeros; fan-out on write naturally produces only the non-zero pairs.
- Hot keys on viral games. The live-session set is a hot key; shard it and count fresh members into a cached total (not an incr/decr counter — passive TTL expiry has no decrement callback). The player bitmap of a viral game is huge but intersects cheaply against a small friend set.
- Exact vs approximate / stale. "50+ friends" and a rounded, few-seconds-stale live count are acceptable and are what unlock caching, sharding, and HyperLogLog. Exactness would forfeit all three.
- TTL-based expiry vs explicit stop. Heartbeat TTL is the correct end-of-session mechanism; explicit stops are an optimization. Depending on stops alone leaks the count upward forever.
- Partition choice. Partitioning graph and has-played by
user_idmakes per-user reads (and fan-out) cheap; the "who played this game" direction is served by the game bitmap instead, not by scanning per-user data.
Extensions if asked
If the interviewer wants to push beyond the core system, add only the extension that changes the design discussion:
- Which friends played (the list, not just the count). Return the actual friend avatars — needs the intersection materialized as a small id list per
(user, game), capped (e.g. top 6 to show, "+9 more"). - Friends playing right now. Intersect the friend set with the game's live session set — the same intersection as the count, but over live sessions instead of has-played facts.
- Trending among your friends. Rank games by friend-played count to power a "your friends are playing" shelf — a periodic batch over the friend-played counters.
- Privacy / blocked users. Exclude blocked or hidden-activity users from both counts — a filter applied at fan-out and at intersection time. There's no dedicated guide yet.
What interviewers look for & common mistakes
What interviewers usually reward:
- Recognizing the friend signal as a set intersection
friends(user) ∩ players(game), and not answering it by looping over friends on every render. - Reaching fan-out on write for the friend count (the news-feed pattern) with a compute-and-cache-on-read fallback for hot users and cold pairs, and knowing the push-vs-pull tradeoff.
- Naming the hot-key problem on a viral game's live-session set and fixing it with a sharded set whose fresh members are counted into a cached total (not an incr/decr counter that can't be decremented on passive expiry).
- Getting session expiry right — heartbeat TTL, not explicit stops — so the live count doesn't leak upward.
- Accepting approximate and eventually-consistent counts ("50+", HyperLogLog for uniques, a few-seconds-stale live count) and using that tolerance to enable caching and sharding.
- Batching both signals for a whole page of games in one request.
Before you finish, do a quick mistake check:
- Did you avoid intersecting friends ∩ players on every render, and instead fan out on write / cache the result?
- Did you avoid a dense table over all (user, game) pairs, storing only non-zero pairs?
- Did you name the viral-game hot key, derive the live count by counting fresh sessions (not incr/decr), and shard the live-session set?
- Did you end sessions by heartbeat TTL rather than trusting explicit stop events?
- Did you use an approximate distinct count (HyperLogLog) for "unique players" instead of an exact giant set?
- Did you treat both counts as derived, rebuildable caches — never the source of truth?
- Did you batch both signals for a page of games in one request?
Practice this live
Run this exact question with our AI voice interviewer and get feedback.
Start the interview →