Design Facebook Post Search

Full-Text Search System Design

Overview

Type "world cup final" into the Facebook search bar and, along with people and pages, you get posts — status updates, comments, shared links — matching those words, ranked so the most relevant and recent surface first. That is a full-text search problem over billions of documents, and it is a genuinely different beast from the search autocomplete system. Autocomplete predicts what you are about to type from a small set of popular query strings and returns them by a prefix lookup; post search takes a completed multi-word query and must find, out of billions of arbitrary free-text posts, the ones whose content actually matches — then rank them. Autocomplete is a prefix-trie problem. Post search is an inverted index problem. If you find yourself designing a trie here, you have answered the wrong question.

It is labeled "Hard" because every naive answer collapses at scale. You cannot scan billions of posts per query — you need an inverted index that maps each word to the posts containing it. That index does not fit on one machine, so you must shard it, and how you shard (by document or by term) changes the entire query path. Returning matches in index order is useless — a query for "coffee" that returns the first billion posts mentioning coffee in insertion order is worthless; you need relevance ranking, and ranking every matching document is far too expensive, so you need a two-phase approach. And a post must become searchable within seconds of being written, which fights directly against the batch-built, immutable index structures that make search fast.

In this walkthrough you'll scope the system, size it, model the data, design the API, draw the read (query) and write (index) paths, and then go deep on the three questions that decide whether the design works: the inverted index and how to shard it, two-phase ranking, and near-real-time indexing.

Out of scope (say so): searching for people/pages/groups (a separate entity-search path), search autocomplete/typeahead (see the linked breakdown), image and video content search, and ads. We own one thing: given a text query, return the most relevant and recent posts the searcher is allowed to see.

Functional requirements

• Keyword post search. Given a free-text query (one or more words), return posts whose text matches, ranked best-first. A multi-word query like "world cup final" should match posts containing those words.
• Ranking by relevance and recency. Results are ordered by a blend of textual relevance, recency, and engagement — not by insertion order and not purely by date.
• Filters. Narrow results by author, time range, and — critically — visibility (the searcher must only see posts they are permitted to see).
• Near-real-time freshness. A newly created post should be searchable within seconds, not minutes or hours.
• Pagination. Return a page of results (e.g. top 20) with a stable cursor for the next page.

Out of scope (state it): people/page/group search, autocomplete, semantic/vector search (extension), media content search, and personalized ML ranking beyond a lightweight social-graph signal.

Non-functional requirements

• Low query latency. p99 well under ~500 ms for a query over billions of posts. Search is interactive; a slow search feels broken.
• Massive scale, read-and-write-heavy. Billions of posts indexed; high query QPS; and a relentless firehose of new posts to index continuously. Unlike a pure read-heavy feed, the write (indexing) path here is a first-class, high-throughput concern.
• Near-real-time indexing. Index freshness measured in seconds. A breaking-news post must be findable almost immediately.
• High availability over strict consistency. A search that misses a post created two seconds ago, or shows a slightly stale ranking, is fine. Returning an error, or timing out, is not.
• Correct visibility filtering. This is a hard correctness requirement, not a nicety — search must never leak a private post to someone who cannot see it.
• Scalable index. The index must shard across many machines and grow horizontally as post volume grows.

Estimations

State assumptions first; the goal is to justify sharding and the two-phase design, not exact arithmetic.

Rounded assumptions

• Total posts indexed: ~2 billion users, but assume a searchable corpus on the order of ~500 billion to 1 trillion posts (many years of history; comments and shares included). Round to ~1 trillion documents.
• New posts/sec (indexing rate): billions of posts + comments per day. ~5B writes/day ÷ 86,400 ≈ ~60K new documents/sec, with a 3–5× peak → ~250K index writes/sec at peak.
• Query QPS: assume ~500M searches/day → ~6K queries/sec average, ~30K QPS at peak with the peak multiplier.
• Average post length: ~30 words after tokenization and stop-word removal → ~15 meaningful indexed terms.
• Index size: the dominant cost is the posting lists. ~1T docs × ~15 terms × ~8–16 bytes per posting (doc id + term frequency + position) ≈ ~150–200 TB raw. Posting-list compression (delta + variable-length encoding) shrinks this by roughly 5–10× to tens of TB stored — still far beyond one machine.
• Latency budget: ~500 ms p99 split as ~network + query parse (~tens of ms) → per-shard candidate retrieval (~tens of ms, run in parallel across all shards) → merge + re-rank top-K (~tens of ms).
• Shard count: divide the stored (compressed) tens of TB by ~tens of GB of index per shard — kept RAM-resident so each shard serves fast random reads into its posting lists — and you land in the hundreds to low thousands of shards. Query fan-out and per-shard latency parallelism push toward the upper end: more shards means each holds a smaller slice and answers faster, so the corpus is split into more shards than storage alone demands, landing around ~1–2K shards. Each shard is replicated ~3× for availability and read spread; shards colocate many-per-node, so the physical fleet is a few thousand nodes, not one node per shard-replica. Shard count is also bounded above by how many partial results the coordinator can merge within budget — another reason each shard returns only its top-K, not raw matches.

How the numbers affect the design

Core entities / data model

The source of truth is the post store; everything the search system holds is a derived index that can be rebuilt from it. Four logical things exist: the raw post, the inverted index (term to posting list), per-document ranking metadata, and the visibility/ACL data used to filter results.

The Post is authoritative. The inverted index is derived — if a shard's index is lost, it is rebuilt by re-tokenizing the posts assigned to that shard. This is the single most important framing to state: the index is a rebuildable performance structure, not a database of record.

DocMeta is deliberately small and colocated with the index shard so the re-ranker can read a candidate's features without a cross-service call on the hot path. Engagement counters are eventually consistent — a ranking computed from a like count that is a few seconds stale is fine.

How the pieces relate. The post store fans out into the derived structures the search system actually queries — the diagram below shows the dependency direction, which is the thing to get right: everything points back to the post store as the rebuildable source.

flowchart TD
    Post["Post (source of truth)\npost_id, author, text,\ncreated_at, visibility"] -->|"tokenize"| Index["Inverted Index\nterm → posting list"]
    Post -->|"extract features"| DocMeta["DocMeta\nengagement, recency, language"]
    Post -->|"extract ACL"| Vis["Visibility\nvisibility, audience_id"]
    Index -.->|"rebuildable from"| Post
    DocMeta -.->|"rebuildable from"| Post
    Vis -.->|"rebuildable from"| Post

API design

Two endpoints carry the product: the user-facing search read endpoint and an internal index write endpoint (called by the indexing pipeline, never by clients).

Search posts

GET /api/search/posts?q=world+cup+final&author=&from=&to=&cursor=&limit=20
200 OK
Returns: {
  "query": "world cup final",
  "results": [
    { "post_id": "...", "author_id": "...", "snippet": "...the world cup final was...", "score": 87.4, "created_at": "..." },
    ...
  ],
  "next_cursor": "..."
}

q is the raw query string (tokenized server-side). author, from, to are optional filters. limit defaults to 20. Pagination is cursor-based, not offset-based — the cursor encodes the position in the ranked result (e.g. the last score + post_id) so newly indexed posts don't shift pages underneath the user. The response returns metadata and a highlighted snippet; it never returns posts the caller cannot see.

A subtlety worth naming: because the index is constantly changing (near-real-time), deep pagination over a ranked result is inherently approximate — a post indexed between page 1 and page 2 can legitimately shift results. The cursor makes each page self-consistent (no duplicates, no skips relative to the ranking snapshot it captures), but interviewers don't expect a perfectly stable deep-scroll over a live index; most searches never paginate past the first page or two anyway.

Index a post (internal, write path)

POST /internal/index
Body: {
  "post_id": "...",
  "author_id": "...",
  "text": "The world cup final was incredible",
  "created_at": "...",
  "visibility": "public",
  "op": "upsert"   // upsert | delete
}
204 No Content

Called by the indexing pipeline when a post is created, edited, or deleted. A create and an edit both arrive as op: upsert — the indexer tombstones the prior occurrence (if any) and indexes the current text into the buffer, so an edit is a delete-plus-reinsert under the hood (deep dive 3). op: delete marks the document as removed. This endpoint is asynchronous relative to user reads and is the high-throughput side of the system (~250K/sec peak).

High-level architecture

The system has a query path (scatter-gather across index shards, then merge and re-rank) and an indexing path (post firehose to tokenizer to inverted-index segments). We'll walk each, then combine.

1. Query path — scatter-gather and two-phase ranking

Every search fans out to all index shards in parallel, each returns its local top candidates, and a coordinator merges and re-ranks them.

flowchart LR
    Client["Client"] -->|"GET /search/posts?q=..."| Coord["Query Coordinator"]
    Coord -->|"parse + tokenize query"| Coord
    Coord -->|"scatter query to all shards"| S1["Shard 1"]
    Coord -->|"scatter"| S2["Shard 2"]
    Coord -->|"scatter"| S3["Shard N"]
    S1 -->|"local top-K candidates"| Coord
    S2 -->|"local top-K candidates"| Coord
    S3 -->|"local top-K candidates"| Coord
    Coord -->|"merge + re-rank top-K + visibility filter"| Result["Ranked page"]
    Result --> Client

1. The query coordinator tokenizes the query into terms (same tokenization used at index time — this must match, or terms won't align: if index-time lowercases and stems "running" to "run" but query-time doesn't, a search for "Running" finds nothing).
2. It scatter-gathers — sending the query to every shard in parallel. Each shard holds a subset of the documents (document-sharding, deep dive 1), so any shard might hold matching posts.
3. Each shard runs phase one locally: intersect the posting lists for the query terms, apply a cheap first-pass score, and return only its local top-K candidates (say top 100) with their features — not all matches.
4. The coordinator gathers the per-shard candidate lists, applies phase two — an expensive re-rank over the merged candidate set — enforces the visibility filter, and returns the final page.

The coordinator is stateless and horizontally scalable; it holds no index, only the routing table of which shards (and replicas) exist. It is also where per-shard timeouts live: if one shard hasn't answered within the budget, the coordinator returns the merged results from the shards that did, accepting a slightly incomplete result rather than stalling the whole query on one slow node.

2. Indexing path — from new post to searchable

A new post flows through a streaming pipeline that tokenizes it and writes it into the inverted index of the shard that owns it.

flowchart LR
    PostSvc["Post Service\n(post created/edited/deleted)"] -->|"change event"| Queue[["Post Firehose\n(Kafka)"]]
    Queue --> Indexer["Indexer Workers\n(tokenize, build postings)"]
    Indexer -->|"route by post_id → shard"| ShardW["Owning Index Shard"]
    ShardW -->|"append to in-memory segment"| MemSeg["In-memory segment\n(fresh, searchable in seconds)"]
    MemSeg -.->|"flush + merge periodically"| DiskSeg["On-disk immutable segments"]

1. Every post create/edit/delete publishes a change event to a firehose (Kafka). This decouples indexing from the write path of the post service.
2. Indexer workers consume events, tokenize the text (lowercase, strip punctuation, remove stop-words, stem), and produce postings { term → post_id, tf, positions }.
3. Each post is routed by post_id to its owning shard (document-sharding). The indexer appends the new postings to that shard's small in-memory segment, which is immediately searchable — this is what delivers seconds-level freshness.
4. In-memory segments are periodically flushed to immutable on-disk segments and merged in the background (deep dive 3).

3. Combined architecture

flowchart TD
    Client["Client"] -->|"search"| Coord["Query Coordinator"]
    Coord -->|"scatter-gather"| Shards["Index Shards\n(inverted index + DocMeta,\ndocument-sharded)"]
    Shards -->|"candidates"| Coord
    Coord -->|"merge + re-rank + visibility"| Client
    PostSvc["Post Service"] -->|"change events"| Kafka[["Post Firehose (Kafka)"]]
    Kafka --> Indexer["Indexer Workers\n(tokenize)"]
    Indexer -->|"route by post_id"| Shards
    PostStore[("Post Store\n(source of truth)")] --- PostSvc
    Shards -.->|"rebuildable from"| PostStore

The query path and indexing path share only the shards, and they don't contend: reads hit immutable segments plus a small in-memory segment; writes append to the in-memory segment and flush in the background. The post store is the source of truth; every shard's index is derived and rebuildable from it.

Two properties fall out of this shape and are worth stating in an interview. First, the two paths scale independently — a spike in query traffic adds coordinator and replica capacity without touching indexers, and a spike in posting volume adds indexer capacity without touching the query path. Second, failure is graceful in both directions — a lagging indexer means posts are searchable a little later (freshness degrades), and a slow or dead shard means a query returns slightly incomplete results (recall degrades), but neither returns an error to the user. That "degrade, don't fail" property is exactly the availability-over-consistency stance the non-functional requirements called for.

Deep dives

1. The inverted index and how to shard it

Tokenization first. Before anything is indexed, a post's raw text is turned into terms: lowercase everything, strip punctuation, split on word boundaries, drop stop-words (the, a, of), and stem ("running", "runs", "ran" → "run") so morphological variants match. The exact same analyzer must run on the query side, or a search won't align with the index. This step is deterministic and cheap, and it is where language handling (extension) plugs in. Get it wrong and no amount of index or ranking cleverness will save the results.

The inverted index. For each term, store a posting list of the documents that contain it. A query for "world cup final" retrieves three posting lists — world, cup, final — and intersects them to find posts containing all three (an AND query), or unions them for OR semantics. Each posting entry carries the post_id, the term frequency, and the term's positions (needed for phrase queries — "world cup" as an adjacent phrase, not just both words somewhere).

flowchart LR
    T1["term: world"] --> P1["postings: p9, p42, p87, p153, ..."]
    T2["term: cup"] --> P2["postings: p42, p87, p200, ..."]
    T3["term: final"] --> P3["postings: p42, p87, p91, ..."]
    P1 --> I["intersect → p42, p87"]
    P2 --> I
    P3 --> I
    I --> R["candidate posts for 'world cup final'"]

This is the whole reason search is fast: instead of scanning a trillion posts, you read a handful of posting lists and intersect them. To keep intersection cheap, posting lists are stored sorted by post_id so two lists can be merged in a linear pass, and they are heavily compressed (delta-encoding the sorted ids, then variable-length encoding) so hundreds of TB shrink substantially and more of each list stays in memory.

Intersection order matters. For an AND query, intersect the shortest posting lists first. A query for "photosynthesis coffee" should start from the rare term ("photosynthesis", a short list) and check those few candidates against the huge "coffee" list — not the other way around. Each posting list carries its own length, and skip-pointers (an index into the compressed list) let the intersection jump ahead past non-matching ids instead of decoding every entry. This is why a multi-word query with one rare term is cheap even when another term is extremely common.

The sharding question. The index is far too large for one machine, so it must be split across thousands of shards. There are two fundamentally different ways to split it, and the choice defines the query path.

Why document-sharding wins. The decisive factor is skew. Term-sharding makes one shard own "the" — billions of postings, hammered by every query that includes a stop-word-adjacent common term — while document-sharding spreads that same term's postings across all N shards, so no single shard is hot. Scatter-gather costs you a fan-out per query, but that is a bounded, parallelizable cost; a hot shard is an unbounded, un-rebalanceable one.

Replication. Each shard is replicated (say 3×) for availability and to spread query load; a replica can serve reads while another is compacting segments. The coordinator can hedge a slow shard by querying a second replica.

Routing at index time. Because a post's shard is hash(post_id) % N, both the indexer (which post goes where) and any point-lookup agree on the owner without a lookup table — the hash is the routing. Growing the fleet means resharding, which is expensive under plain modulo hashing (changing N remaps most posts); production systems use a fixed large number of logical shards mapped onto physical nodes, so scaling moves whole logical shards between nodes instead of rehashing every document. Worth a sentence in an interview to show you've thought past the happy path of a static cluster.

2. Ranking — why index order is useless and how two-phase works

Why you can't return posting-list order. After intersecting posting lists you have a set of matching post_ids — but in what order? If posting lists are sorted by post_id, "index order" is essentially insertion order, which is meaningless to a searcher. A query for "coffee" would return the oldest (or arbitrary) posts mentioning coffee, not the best ones. Search's real job is ranking: ordering matches by how well they satisfy the query.

The ranking signals. A good post score blends several factors:

• Textual relevance — how well the post's text matches the query terms, scored with TF-IDF or BM25. The idea: a term is more meaningful if it appears often in this post (TF) but rarely across all posts (IDF) — so matching "photosynthesis" counts far more than matching "the". BM25 refines plain TF-IDF with two corrections that matter here: term-frequency saturation (via k1) — the 10th occurrence of a term adds far less than the 2nd, so a post that repeats "coffee" 50 times doesn't run away with the score — and document-length normalization (via b), so a long rambling post isn't unfairly favored over a crisp short one just for containing the term more times.
• Recency — newer posts usually rank higher, via a time-decay factor. A post from an hour ago about "world cup final" beats one from four years ago.
• Engagement — likes, comments, shares as a quality/popularity signal.
• Social-graph proximity — posts by the searcher's friends or followed pages rank higher than posts by strangers (a lightweight personalization signal).

These combine into a single score, e.g. score = w1 · BM25 + w2 · recency_decay + w3 · log(engagement) + w4 · graph_proximity, with weights tuned offline against labeled relevance data. The exact formula matters less in an interview than naming the categories of signal and being clear about which are static (bakeable into the index) versus dynamic (must be computed at query time) — that split is what drives the two-phase design below.

A subtlety of document-sharding: local IDF. Because each shard scores with its local collection statistics (its own document count N and per-term document frequency df), IDF is computed per-shard and is therefore slightly inconsistent across shards — the same term looks marginally rarer on one shard than another. At this scale shards are statistically similar (random post_id hashing gives each a representative sample), so it rarely perturbs the top results. When it does matter, engines offer a distributed-frequencies mode (Elasticsearch's dfs_query_then_fetch) that gathers global term stats before scoring, trading an extra round trip for globally-consistent IDF.

The problem: ranking every match is too expensive. A common query can match hundreds of millions of posts. Computing a full BM25 + recency + engagement + social-graph score for every match, then sorting, is far beyond the latency budget. The fix is two-phase retrieval.

flowchart TD
    Q["query terms"] --> Phase1
    subgraph Phase1["Phase 1 — per shard, cheap"]
        R1["intersect posting lists"] --> R2["cheap first-pass score\n(BM25 + light recency)"]
        R2 --> R3["keep local top-K\n(e.g. top 100)"]
    end
    Phase1 --> Merge["Coordinator gathers\nN × top-K candidates"]
    Merge --> Phase2
    subgraph Phase2["Phase 2 — merged set, expensive"]
        E1["full re-rank\n(BM25 + recency + engagement + social graph)"] --> E2["visibility filter"]
        E2 --> E3["final sorted page"]
    end

Two-phase retrieval works like this:

1. Phase one (candidate retrieval, per shard, cheap). Each shard intersects the query's posting lists and applies a cheap first-pass score — typically BM25 with a light recency term, using only data already in the posting list and local DocMeta. Each shard keeps only its local top-K (say top 100) and returns those. This phase optimizes for recall — don't drop good documents — while being fast enough to run over huge posting lists.
2. Phase two (re-rank, on the merged set, expensive). The coordinator gathers N shards × top-K candidates — a few thousand posts, not hundreds of millions — and runs the full expensive scoring (BM25 + recency decay + engagement + social-graph proximity, possibly an ML model). Because it operates on a few thousand candidates instead of the whole match set, an expensive per-document computation is now affordable. Then it applies the visibility filter and returns the top 20.

The insight is the funnel: phase one cheaply reduces billions of matches to thousands of good candidates across all shards; phase two spends real compute only on those thousands. You get near-optimal ranking at a fraction of the cost of scoring everything.

Why this doesn't hurt quality much. The risk of a funnel is dropping a truly great result in phase one before phase two ever sees it. Two things keep this safe: phase one's cheap score (BM25 + light recency) is already well-correlated with the full score, so the genuinely best posts almost always survive; and each shard returns a generous top-K (100, not 20), so the candidate net is wide. The rare case where a post that would have ranked #5 under the full model gets cut because it was #150 on its shard is an acceptable quality trade for the massive latency win. If measured recall suffers, widen the per-shard top-K — the knob is right there.

A note on AND-intersect vs scored disjunction. The retrieval so far is framed as AND-intersecting posting lists — the right mental model, and correct for strict phrase/AND queries. But real relevance-ranked search runs a scored disjunction (OR): a doc matching two of three query terms can still be a strong result, so the engine scores the union rather than requiring every term. The point of WAND below is precisely to make that disjunctive, scored retrieval fast — it accelerates the OR case that pure AND-intersection would otherwise miss.

Early termination. Phase one can stop early using techniques like WAND (Weak AND): maintain a running threshold of the K-th best score seen so far, and skip any document that can't beat it. What makes the skip safe is that each posting list carries a precomputed max-term-contribution (the largest score that term can add to any document). Summing those upper bounds gives a document's maximum possible score without fully decoding and scoring it; if that upper bound can't exceed the current K-th best, the document is skipped entirely. On a common term this prunes most of the posting list — another reason phase one stays fast even when a term matches hundreds of millions of posts.

Sort-by-recency filter. Some queries want newest-first ("latest" tab). Snowflake ids are only approximately reverse-chronological: they are time-ordered to the millisecond, but the per-node sequence bits break within-millisecond ties by node, so id order is not a strict global time order — strict newest-first would sort by created_at as the primary key. The important structural point is that the post_id-sorted posting lists used for AND-intersection are the wrong lists for a recency query: they're ordered for fast intersection, not for time traversal. A recency-first query needs a separate posting-list variant per term, sorted by created_at DESC (or Snowflake id), so the query walks the newest matches first and stops as soon as it has a page — no full re-rank needed. Maintaining this second ordering is the cost of a fast "latest" tab; a time-sortable post id keeps that variant cheap to build and lets cursor pagination fall out of the id itself.

3. Near-real-time indexing

The tension. Search is fastest over large, sorted, compressed, immutable index files — but a post must be searchable within seconds of being written, and the firehose delivers ~250K writes/sec at peak. You cannot rebuild a hundred-TB immutable index per new post. Nor can you mutate a posting list in place cheaply: inserting one id into a compressed, sorted, on-disk list means rewriting the block. The resolution is the segment model that Lucene (and thus Elasticsearch/Solr) uses — never mutate, only append new immutable pieces and merge them later.

Immutable segments plus a live in-memory buffer. The index for a shard is not one monolithic file — it is a set of immutable segments, each a self-contained mini inverted index over a batch of documents. New posts don't modify any existing segment. Instead:

flowchart LR
    New["new post"] --> Buf["in-memory segment\n(writable buffer)"]
    Buf -->|"refresh every ~1s"| Small["small immutable segment\n(searchable)"]
    Small -->|"background merge"| Med["merged segment"]
    Med -->|"merge again"| Big["large segment"]
    Query["query"] -.->|"searches ALL segments\n+ in-memory buffer"| Buf
    Query -.-> Small
    Query -.-> Med
    Query -.-> Big

1. New postings are appended to a small in-memory writable buffer for the shard.
2. On a short refresh interval (~1 second) the buffer is sealed into a new small immutable on-disk segment and becomes searchable. This ~1s refresh is what delivers near-real-time freshness.
3. A query searches every segment for the shard — all immutable segments plus the live in-memory buffer — and merges their per-term postings. More segments means more lists to merge, which is why...
4. A background merge process periodically combines many small segments into fewer larger ones (like LSM-tree compaction). Merging keeps the segment count bounded so query-time merging stays cheap, and it is where compression and dead-document purging happen.

Deletes and edits against an append-only index. You can't remove a posting from an immutable segment. So a delete just marks the document in a per-segment deleted-docs bitset (tombstone); queries filter out tombstoned documents at read time. Crucially, the tombstone is scoped to the old document's occurrence in its old segment — Lucene tombstones the internal doc number (the document's slot within that specific segment), not the external post_id — the indexer finds that old slot by looking up the post_id (itself an indexed field) across the shard's live segments. An edit is therefore a delete-plus-reinsert: tombstone the old document's slot in its old segment, and index the new version as a fresh entry in the current in-memory buffer. Only the reinserted version is live; the old one is masked. The space held by tombstoned documents is only actually reclaimed during a background merge, which physically drops them from the merged output.

This lazy-delete model has a consequence worth stating: for a while after an edit, the old version's postings still physically exist in an old segment and the new version's postings exist in the buffer, but only the new one is live — the old segment slot is tombstoned. Search correctness comes from the tombstone filter, not from the postings being gone. This is exactly the LSM-tree read-path pattern — merge live data across levels, skip the deleted keys — applied to an inverted index.

The freshness-vs-throughput tradeoff. The refresh interval is the central knob:

Durability. The in-memory buffer would lose recent posts on a crash, so each shard also appends incoming postings to a write-ahead / translog on disk before acknowledging. On restart the buffer is replayed from the translog. And because the post store is the source of truth, a totally lost shard can be rebuilt by re-indexing its posts from scratch.

Ordering and idempotency in the firehose. These are two separate concerns. Ordering: partition the firehose by post_id so all events for one post land on the same partition and are consumed by one indexer in order — so a delete that supersedes a create isn't undone by a late-arriving create. Partitioning alone gives ordering; it does not remove duplicates. Idempotency: Kafka delivers at-least-once, so the same event can arrive twice — the indexer must be idempotent, keying the buffer by post_id so a re-delivered upsert overwrites rather than appends (no doubled postings) and using the post's version (or created_at/edit timestamp) to discard an event older than what's already indexed.

Tradeoffs & bottlenecks

• Document sharding vs term sharding. Document-sharding balances load and keeps ranking features local, at the cost of scatter-gathering every query to all shards. Term-sharding avoids fan-out for single-term queries but concentrates hot terms on one shard and ships giant posting lists across the network. Document-sharding wins at scale; pair it with two-phase ranking to keep the gather cheap.
• Index-time vs query-time ranking. Some ranking signals (a static quality/authority score, language) can be baked into the index at write time so phase-one scoring is nearly free; others (recency, live engagement, social-graph proximity) must be applied at query time because they change after indexing. The split: cheap, static signals in phase one; dynamic, personalized signals in phase two.
• Freshness vs merge cost. A shorter refresh interval means fresher results but more small segments, higher query-time merge cost, and more background-merge CPU/IO. ~1 second is the standard balance; it is a per-index tuning knob, not a global constant.
• Precision vs recall. Phase one favors recall (keep a wide candidate net so no good post is dropped) and is cheap; phase two favors precision (rank the candidates well) and is expensive. Getting the phase-one top-K size right matters: too small drops relevant posts before re-ranking (hurts recall); too large makes the gather and re-rank expensive.
• Tail latency from scatter-gather. A query is only as fast as its slowest shard. Replicas, per-shard timeouts (return partial results rather than stall), and request hedging bound the tail.
• Visibility filtering cost. Filtering out posts the searcher can't see is a hard correctness requirement and adds work to every query. Doing it in phase two on the small candidate set (rather than mid-retrieval on billions) keeps it affordable — at the cost of occasionally retrieving candidates that get filtered out, so phase one must over-fetch a little.
• Index compression vs decode cost. Aggressive posting-list compression (delta + variable-length encoding) shrinks the index so more of it fits in memory and less is read from disk — but every query pays CPU to decode. Skip-pointers let intersection decode only the stretches it actually needs. The balance favors heavy compression at this scale because I/O and memory are the binding constraints, not CPU.
• Colocating DocMeta on the shard vs a central feature store. Keeping ranking features next to the postings makes phase-one scoring a local, no-network operation — the whole reason scatter-gather is fast — but it duplicates and must keep in sync features that also live elsewhere. A central feature store would avoid duplication but add a network hop per candidate. Colocation wins on the hot path.
• Relevance ordering vs recency ordering. The default "top" tab and the "latest" tab want different orderings, and one posting-list layout can't serve both cheaply: relevance retrieval wants lists sorted by post_id for fast intersection, while a recency tab wants lists sorted by created_at DESC to walk newest-first and stop early. Supporting both means maintaining a second posting-list ordering per term — extra index size and write cost for the "latest" feature. Skip it if the product only needs relevance ranking.

Extensions if asked

Add only the extension that changes the design discussion:

• Personalization / social-graph ranking. Boost posts from the searcher's friends and followed pages. Implemented as a phase-two feature — fetch the searcher's graph (cached) and score graph proximity during re-rank. Full ML ranking (a learning-to-rank model over hundreds of features) is a natural evolution of phase two — it fits there precisely because the candidate set is already small, so an expensive model runs on thousands of docs, not billions.
• Typo tolerance / fuzzy matching. When query terms match little, expand them to near-spellings via an edit-distance or n-gram index. The n-gram concept is shared with the fuzzy-matching approach in the search autocomplete breakdown, but the index structure differs: here n-grams index the corpus's document terms (to find similarly-spelled words to expand a query to), whereas autocomplete n-grams index suggestion strings. Do this at query-parse time, adding expanded terms to the retrieval.
• Phrase and proximity queries. "world cup final" as an exact adjacent phrase uses the positions stored in each posting entry. Intersect the posting lists to find docs containing all three terms, then for each surviving doc check the positions line up: some position of cup equals a position of world + 1, and final equals that position + 1 (proximity queries relax "+1" to "within N"). This is why postings store term positions, not just doc ids.
• Privacy / visibility at query time. For complex audiences (friends-of-friends, group membership), fold an ACL check into phase two: attach the audience scope to each candidate's DocMeta and evaluate it against the searcher's identity before returning. Never rely on ranking to hide a post — it must be a hard filter.
• Multi-language. Language-specific tokenization, stemming, and stop-word lists; detect a post's language at index time and tokenize accordingly, and analyze the query in the searcher's language. A CJK language needs word-segmentation rather than whitespace splitting, so the analyzer, not the index structure, is what changes.
• Semantic / vector search. Beyond exact-keyword matching, embed posts and queries into a vector space and retrieve by nearest-neighbor (ANN) to catch paraphrases and synonyms. This is a parallel retrieval path whose candidates merge into phase two alongside the inverted-index candidates — a hybrid lexical + semantic retriever. Mention it; don't design the ANN index in this interview.

What interviewers look for & common mistakes

This question rewards candidates who recognize it as a search-engine problem and reach for the right primitives rather than reinventing them. The strongest answers make three moves early: name the inverted index as the only viable retrieval structure, choose document-sharding with a crisp reason (hot-term skew), and separate retrieval from ranking via two phases. Everything else — freshness, visibility, compression — builds on those three.

What interviewers usually reward:

• Reaching for an inverted index immediately — recognizing this is full-text search, not a trie/prefix problem, and never proposing to scan posts.
• Choosing document-sharding and justifying it against term-sharding via the hot-term skew argument, and naming scatter-gather as the resulting query pattern.
• Two-phase ranking — cheap per-shard candidate retrieval for recall, expensive re-rank on the merged top-K for precision — and explaining why single-phase full ranking is infeasible.
• The segment / merge model for near-real-time indexing — immutable segments + in-memory buffer + ~1s refresh + background merge + tombstone deletes — and articulating the freshness-vs-throughput tradeoff.
• Treating the index as derived and rebuildable from the post store, with a translog for crash durability.
• Visibility as a hard filter, applied in phase two on the candidate set, never leaking a private post.

Before you finish, do a quick mistake check:

• Did you use an inverted index (term → posting list) and never propose scanning posts?
• Did you shard by document, not by term, and explain the hot-term problem term-sharding causes?
• Did you name scatter-gather and two-phase retrieval, and explain why ranking every match is impossible?
• Did you rank by relevance (BM25/TF-IDF) + recency + engagement + social graph, rather than returning posting-list order?
• Did you make new posts searchable in seconds via the segment model (in-memory buffer + ~1s refresh + immutable segments + background merge), and handle deletes with tombstones?
• Did you state the freshness-vs-throughput tradeoff around the refresh interval?
• Did you enforce visibility as a hard filter so search never leaks a post the searcher can't see?