Design a News Aggregator

Google News System Design

Overview

A news aggregator like Google News does something a plain feed does not: it reads the whole press — thousands of publishers writing about the same events — and collapses that flood into a clean, ranked page of stories. You open it and see "one entry per event," each backed by dozens of sources, freshest and most important first, tuned to what you read. The hard part isn't fetching articles; it's realizing that fifty outlets just published the same event and showing it once.

That collapsing step is what separates a strong answer from a shallow one. A naive design crawls sources, dumps every article into a list, sorts by time, and serves it — and the result is unreadable: twenty near-identical wire-service reprints of one press release, stacked on top of each other, burying the next story. The product's core value is the opposite of a raw feed: deduplication of reprints and clustering of independent coverage into a single canonical story.

It's labeled "Medium" because each moving part borrows a well-understood pattern — but the combination is the challenge, and one piece (clustering) has no off-the-shelf answer. You crawl like a web crawler, but with a freshness budget of minutes, not days. You dedup with fingerprints like post search uses, but must also group differently-worded coverage of one event. And you rank and serve read-heavy like a news feed, but the unit is a story, not a post. This walkthrough reuses those systems and cross-references them rather than re-deriving them.

In this walkthrough you'll scope the aggregator, size it, model the data, sketch the API, draw the ingest and serving paths, then go deep on the three parts that decide whether it works: crawl + ingest, story clustering, and ranking + personalization + serving.

Out of scope (say so): writing or editing the articles (we aggregate others' content, we don't author it), the full-text search over articles (a separate search system), comment threads, and the ad/monetization layer. We own discovering, deduplicating, clustering, ranking, and serving third-party news.

Functional requirements

• Crawl news sources. Continuously discover and poll many publishers — via RSS feeds, sitemaps, and direct crawling — and extract clean article text and metadata.
• Deduplicate and cluster. Detect exact and near-duplicate reprints, and group articles about the same event into one story, even when written in different words.
• Pick a canonical story. For each cluster, choose a representative headline/article and attach the rest as "related coverage."
• Rank stories. Order stories by a blend of freshness, importance (how many sources are covering it, source authority), and topical fit — not raw publish time.
• Personalize. Bias the feed toward topics, sources, and regions the user reads, without hiding major breaking news.
• Serve a fresh feed with low latency. Return a paginated, ranked feed (and topic/section pages) fast, updated within minutes of new coverage.

Out of scope (state it): authoring content, full-text search, comments, and ads. This system ingests, clusters, ranks, and serves — it does not create news.

Non-functional requirements

• Freshness in minutes, not days. This is the headline constraint that separates a news crawler from a general web crawler. A breaking story that appears an hour late has failed; target new coverage visible within a few minutes of publication.
• Read-heavy serving. Feed reads vastly outnumber article ingests. The serving path must be cache-friendly and low-latency (feed load well under ~300 ms p99), like a news feed.
• Clustering quality over perfection. Grouping is inherently fuzzy; a good answer favors high precision (don't merge two different events into one story) while accepting occasional missed merges. A wrong merge is far more visible than a missed one.
• Politeness to sources. Crawling must respect robots.txt and per-host rate limits — the crawler cannot hammer a publisher (see the web crawler).
• Eventual consistency is fine. A story appearing a minute late, or a cluster gaining a source a few seconds later, is acceptable. Availability beats strict consistency.
• Extensible. New sources, topics, languages, and quality signals should be addable without a redesign.

Estimations

State assumptions; the goal is to justify the ingest rate, the clustering cost, and the read-heavy serving path, not to be exact.

Rounded assumptions

• Sources: ~100K news sites/feeds worldwide worth polling. Most publish a handful of articles a day; a few hundred publish constantly.
• Articles ingested/day: ~10M new articles (including reprints). ÷ 86,400 s ≈ ~120 articles/sec average, call it a few hundred/sec at peak (a big news day). This is the write/ingest rate.
• Stories/day: ~1M distinct events. So on the order of ~10 articles per story on average — but hugely skewed: a major event draws thousands of articles, a niche one draws two. That ~10:1 collapse is exactly the value the product adds.
• Users: ~100M DAU, each opening the feed several times a day → ~500M feed reads/day ≈ ~6K reads/sec average, ~2–3× at peak. Read:write ≈ 50:1 (500M reads vs 10M ingests) — decisively read-heavy.
• Freshness poll cadence: high-value sources polled every ~1–2 minutes; the long tail every ~15–30 min. Polling 100K sources on a fixed 1-min cadence would be ~1,600 polls/sec of mostly-unchanged feeds — so poll frequency is adaptive (deep dive 1).

How the numbers affect the design

Core entities / data model

The source of truth is the article store (raw fetched content). Everything else — clusters, rankings, per-user feeds — is derived and rebuildable from it, the same framing as the derived index in post search.

The Article is authoritative; content_hash and simhash are dedup fingerprints (deep dive 2), and embedding is the semantic vector used for same-event clustering. The Story is the derived cluster: it holds its members, the chosen canonical article, a rolling source_count (a key importance signal — how many independent outlets are covering it), and a centroid_embedding used to test whether a new article belongs. UserProfile drives personalization (deep dive 3).

The dependency direction is the thing to get right: the article store is the rebuildable source of truth, and stories, rankings, and per-user feeds all derive from it — the same framing as the derived index in post search.

flowchart TD
    Article["Article (source of truth)<br/>text, url, published_at"] -->|"fingerprint"| FP["content_hash + simhash"]
    Article -->|"embed"| Vec["embedding vector"]
    FP -->|"dedup + near-dup"| Story["Story (cluster)<br/>members, source_count,<br/>canonical, centroid"]
    Vec -->|"semantic cluster"| Story
    Story -->|"score"| Rank["Ranked story list<br/>(global, cached)"]
    Rank -->|"personalize"| Feed["Per-user feed"]
    Story -.->|"rebuildable from"| Article
    Rank -.->|"rebuildable from"| Article

API design

Two public read endpoints carry the product (feed and story detail); ingestion is an internal pipeline, not a client-facing API, so we describe its component contract rather than HTTP routes.

Get the feed

GET /api/feed?topic=&region=&cursor=<opaque>&limit=20
200 OK
Returns: {
  "stories": [
    { "cluster_id": "...", "headline": "...", "canonical_url": "...",
      "source_count": 34, "top_sources": ["Reuters","AP","BBC"],
      "published_at": "...", "score": 91.2 },
    ...
  ],
  "next_cursor": "<opaque>"
}

The hot, read-heavy path. Returns a ranked page of stories (one entry per event, not per article), optionally scoped to a topic/region section, personalized for the caller. Pagination is cursor-based, not offset — new stories constantly shift positions, so an opaque cursor keeps pages stable (same reasoning as the news feed). The response carries only what the card needs (headline, source count, top sources); article bodies are fetched on click.

Get one story (cluster detail)

GET /api/story/{cluster_id}
200 OK
Returns: {
  "cluster_id": "...", "canonical": { article... },
  "coverage": [ { source, title, url, published_at }, ... ],  // related articles
  "source_count": 34, "timeline": [...], "topic": ["politics"]
}

Expands one cluster: the canonical article plus the related coverage from other sources. This is the payoff of clustering — "the same event, from 34 outlets, in one place."

Ingest contract (internal)

The ingest pipeline is an internal chain, so what matters is the contract between stages, not a REST route:

ingest(raw_page) -> void
  Effect: extract clean text + metadata → canonicalize URL →
          dedup(content_hash, simhash) → if new, embed → cluster(embedding) →
          assign cluster_id, update Story (members, source_count, canonical) →
          mark the story's ranking dirty for re-scoring.

High-level architecture

Two paths carry the system: an ingest path (sources → clean articles → dedup → clusters → ranked stories) and a serving path (read a ranked, personalized feed of stories). We'll walk each, then combine.

1. Ingest path

New articles flow from sources through extraction, dedup, clustering, and into materialized, ranked stories.

flowchart LR
    Src["News sources<br/>(RSS / sitemap / crawl)"] -->|"poll (adaptive)"| Crawl["Crawler / Poller"]
    Crawl -->|"raw page"| Extract["Extractor<br/>(clean text + metadata)"]
    Extract -->|"canonical URL + text"| Dedup{"Dedup<br/>(content_hash → simhash)"}
    Dedup -.->|"exact/near dup → attach, stop"| Store[("Article Store")]
    Dedup -->|"new article → embed"| Embed["Embedding model"]
    Embed -->|"vector"| Cluster["Incremental clusterer<br/>(ANN over story centroids)"]
    Cluster -->|"assign cluster_id"| StoryDB[("Story Store<br/>(clusters + members)")]
    StoryDB -->|"mark dirty"| Rank["Ranking job<br/>(freshness + importance)"]
    Rank --> Feed[("Ranked story cache")]
    Extract --> Store

1. The crawler/poller pulls each source on an adaptive cadence — fast for prolific/high-value sources, slow for the long tail (deep dive 1) — via RSS/sitemap where available, direct crawl otherwise, respecting robots.txt and per-host limits.
2. The extractor turns the raw HTML into clean article text plus metadata (title, publish time, author), stripping boilerplate (nav, ads).
3. Dedup canonicalizes the URL and checks fingerprints: an exact content_hash or near-duplicate simhash match means "this is a reprint" — attach it to the existing story and stop (no re-embed, no re-cluster).
4. A genuinely new article is embedded into a vector, then handed to the incremental clusterer, which finds the nearest existing story centroid; if close enough it joins that cluster, else it seeds a new one (deep dive 2).
5. Joining a cluster updates the Story (members, source_count, maybe a new canonical pick) and marks it dirty so the ranking job re-scores it. Ranked stories land in a cache the serving path reads.

2. Serving path

A feed read serves precomputed ranked stories, personalizes over a candidate set, and returns a page.

flowchart LR
    Client["Client"] -->|"GET /feed"| FeedSvc["Feed Service"]
    FeedSvc -->|"read top ranked stories<br/>(global + section)"| Cache[("Ranked story cache")]
    FeedSvc -->|"read affinities"| Prof[("User Profile")]
    FeedSvc -->|"re-rank candidates<br/>with personalization"| FeedSvc
    FeedSvc -->|"hydrate headlines"| StoryDB[("Story Store")]
    FeedSvc -->|"story page"| Client

1. The Feed Service reads the top globally-ranked stories (and any requested section) from the ranked story cache — the cheap common case, precomputed by the ingest-side ranking job.
2. It reads the caller's affinities from the User Profile and re-ranks the candidate set (a few hundred stories, not the whole corpus) with a personalization boost.
3. It hydrates the headline/card fields for the returned page and returns one page with an opaque cursor.

The split mirrors the news feed: expensive global ranking is precomputed on the write side; cheap per-user personalization happens at read time over a small candidate set.

3. Combined architecture

flowchart LR
    Src["Sources"] -->|"adaptive poll"| Crawl["Crawler / Poller"]
    Crawl --> Extract["Extractor"]
    Extract --> Store[("Article Store")]
    Extract --> Dedup{"Dedup (hash + simhash)"}
    Dedup -->|"new"| Cluster["Embed + incremental cluster"]
    Cluster --> StoryDB[("Story Store")]
    StoryDB --> Rank["Ranking job<br/>(freshness + importance)"]
    Rank --> Cache[("Ranked story cache")]
    Client["Client"] --> FeedSvc["Feed Service"]
    FeedSvc --> Cache
    FeedSvc --> Prof[("User Profile")]
    FeedSvc --> StoryDB
    StoryDB -.->|"rebuildable from"| Store

Ingest and serving are decoupled: an ingest burst (a big news day) hits the crawler, clusterer, and ranking job, never the read path. Reads serve precomputed ranked stories with personalization layered on. The Article Store is the source of truth; the Story clusters and rankings are derived and rebuildable from it — so a lost cluster index costs a re-cluster, not data loss.

Deep dives

1. Crawl + ingest — freshness, extraction, and exact dedup

The write side of the system: discover sources, poll them fast enough for news, extract clean text, and reject exact reprints before they reach clustering. The mechanics of polite, distributed crawling — the URL frontier, per-host rate limits, robots.txt, DNS caching, crawler traps — are exactly the web crawler's deep dives, so reuse that system rather than re-deriving it. What's different for news is freshness and structured discovery.

Discovery: prefer feeds over blind crawl. A general crawler discovers pages by following links. A news aggregator has a huge shortcut: publishers advertise their new articles.

• RSS/Atom feeds list a source's latest articles with title, link, and publish time — the cheapest, freshest discovery signal. Polling one small feed tells you everything new from that source.
• News sitemaps (sitemap.xml, and the Google-News sitemap variant) list recent URLs with timestamps for sources that expose them.
• Direct crawl of section pages (the homepage, /politics) is the fallback for sources with no feed — link-extract like the web crawler, but scoped to that site.

Freshness: poll adaptively, in minutes. News is worthless late, but polling 100K sources every minute is mostly wasted requests against feeds that rarely change. The fix is adaptive polling: each source has a poll_interval that tightens when it publishes often and relaxes when it's quiet — the same change-frequency scheduling the web crawler uses for re-crawls, but with a minutes-scale floor instead of days.

Extraction: clean text out of messy HTML. A fetched page is mostly boilerplate — nav bars, ads, related-links rails, cookie banners. The extractor isolates the article body (readability-style heuristics or a learned content extractor), pulls structured metadata (headline, author, publish time — often from <meta>/JSON-LD tags publishers embed), and normalizes it. Clean text matters downstream: dedup fingerprints and clustering embeddings computed over navigation junk are noise.

Exact dedup at ingest. The same article often arrives multiple times — the same URL re-fetched, or a wire story on ?utm_source= variants. Two cheap guards, borrowed straight from the web crawler:

• URL canonicalization collapses trivially-different URLs (strip tracking params, fragments, default ports) so the same address isn't processed twice.
• Exact content hash over the clean text: if a newly-extracted article's content_hash matches one already stored, it's an exact reprint — attach it to that article's story (bumping source_count) and stop. No re-embed, no re-cluster.

Exact dedup catches verbatim reprints. The harder cases — lightly-edited reprints and differently-worded coverage of one event — are the next deep dive.

2. Story clustering — near-duplicate detection + same-event grouping

This is the crux and the part with no off-the-shelf answer. The goal: given a stream of new articles, group every article about the same real-world event into one story — collapsing that ~10:1 article-to-story ratio. Two distinct sub-problems stack here, and they need different tools.

Sub-problem A — near-duplicate reprints (same words, minor edits). A wire-service story reprinted by fifty outlets is nearly byte-identical but differs in a headline tweak, a local dateline, or trailing ads — so the exact content_hash from deep dive 1 misses it. This is textbook near-duplicate detection, and the tools are the same fingerprints the web crawler and post search reference:

• Shingling + MinHash / SimHash. Turn each article into a fingerprint such that near-identical texts get near-identical fingerprints. SimHash in particular maps a document to a bit-vector where a small edit flips only a few bits — so "are these near-duplicates?" becomes "is the Hamming distance between their SimHashes below a threshold?"
• Index, don't scan. You cannot compare a new article's fingerprint against all ~1M live stories. Store SimHashes so that near-fingerprints are found by lookup — bucket by fingerprint prefixes/bands (LSH) so a query returns only candidates within a few bits, then verify. This is the same "fingerprint index, not a scan" lesson as the crawler's seen-set.

A near-duplicate hit means "this is a reprint of an article already in story S" → attach to S, bump source_count, done.

Sub-problem B — same event, different words (semantic clustering). The genuinely hard case: two outlets independently write original articles about the same earthquake. The words differ, so no fingerprint matches — yet they're the same story. This needs semantic similarity, not lexical:

• Embeddings. Run each article's clean text (title + lead paragraphs carry most of the signal) through an embedding model into a vector, so two articles about the same event land near each other in vector space regardless of wording.
• Cluster by vector proximity. A story is a cluster of nearby vectors with a centroid. A new article's embedding is compared to existing story centroids; if it's within a similarity threshold of the nearest one (and time-compatible), it joins that story, else it seeds a new story.

Incremental clustering — the streaming constraint. Articles arrive continuously (~120/sec), and there are ~1M live stories. You cannot re-run a batch clustering over everything per article. So clustering is incremental / online:

flowchart TD
    A["new article"] --> H{"exact content_hash<br/>match?"}
    H -->|"yes"| Attach["attach to that story<br/>(bump source_count)"]
    H -->|"no"| S{"SimHash near-dup<br/>within threshold?"}
    S -->|"yes"| Attach
    S -->|"no"| E["embed article"]
    E --> NN["ANN lookup:<br/>nearest story centroid<br/>(recent window)"]
    NN --> C{"similarity ><br/>threshold?"}
    C -->|"yes"| Join["join story:<br/>add member, update centroid"]
    C -->|"no"| New["seed new story<br/>(centroid = this vector)"]
    Join --> Rank["mark story dirty for re-rank"]
    New --> Rank
    Attach --> Rank

1. Exact hash → attach (deep dive 1). Cheapest, tried first.
2. SimHash near-dup within threshold → attach. Catches reprints.
3. Otherwise embed and do an approximate-nearest-neighbor (ANN) lookup over story centroids — restricted to a recent window keyed off each story's last_updated_at, not first_seen_at. A story that is still drawing coverage stays "live" and searchable for as long as articles keep joining, so an ongoing story (a trial, a war) doesn't fragment into a "day 1" and a "day 30" cluster; only genuinely dormant stories age out of the window. This is what keeps clustering sub-linear instead of scanning 1M stories.
4. If the match is close enough, join and update the centroid; else seed a new story. Either way, mark the story dirty for re-ranking. Match against multiple recent members, not just the centroid — a centroid drifts as a cluster grows and will otherwise widen what it accepts, letting two distinct events bridge-merge through tangential articles. Cap each cluster's radius (a tightness threshold) so it can't grow unboundedly inclusive; this is how you actually deliver the "favor precision" promise below.

The two-layer structure is the whole trick: cheap fingerprints (hash → SimHash) resolve the large fraction of articles that are verbatim or near-verbatim reprints (assume most syndicated wire copy), so the expensive embedding + ANN step only runs on the remaining, genuinely differently-worded articles. (Don't over-claim a specific percentage — the 10:1 article-to-story collapse doesn't imply 90% are fingerprint-catchable reprints; a story of 10 members can be 3 wire copies plus 7 independently-written pieces that still need the semantic layer.)

Picking the canonical article. A cluster needs one headline. Choose the canonical by a small heuristic: prefer a high-authority source, an original (non-reprint) report, a complete article (not a stub), and reasonable recency. The rest become "related coverage." Re-evaluate as the cluster grows — an early wire snippet may be superseded by a fuller national-outlet write-up.

3. Ranking + personalization + serving — a fresh, read-heavy feed

The read side. Stories are the unit; the job is to order them by a blend of freshness and importance, personalize per user, and serve it all fast under a 50:1 read:write ratio. The feed-serving machinery — precompute-then-serve, candidate-generate → score → order, cursor pagination, hot-key caching — is the news feed's territory, so reuse it; what's news-specific is the ranking signals and the freshness/stability tension.

Ranking signals — freshness + importance, not raw time. A story's global score blends:

• Freshness — a time-decay on the story's latest activity. News decays fast; a story loses rank over hours, and a fresh burst of new coverage can revive it.
• Importance via coverage volume — source_count is a powerful signal: an event covered by 200 independent outlets is objectively bigger than one covered by 2. This is a signal only clustering can give you — another reason clustering is the core of the system.
• Source authority — coverage from high-authority outlets counts more than from content farms (also a quality/misinformation guard — see extensions).
• Velocity / breaking — the rate at which sources are joining a cluster detects a breaking story before its absolute count is high.

story_score = w1·freshness_decay + w2·log(source_count) + w3·avg_source_authority + w4·velocity. This global score is computed on the ingest side by the ranking job whenever a story is marked dirty, and cached — so reads don't recompute it.

flowchart LR
    Stories["all live stories"] -->|"global score<br/>(precomputed on ingest)"| Top["top-N ranked candidates<br/>(shared, cached)"]
    Prof["user affinities<br/>(topic / source / region)"] --> Boost["personalization boost"]
    Top --> Boost
    Boost -->|"re-rank candidate set"| Page["top 20 stories<br/>(this user's page)"]
    Page -->|"cursor pagination"| Client["Client"]

Personalization — bias, don't replace. On top of the global ranking, boost stories matching the user's topic/source/region affinities (from UserProfile). Crucially, personalization is a re-rank of a candidate set, exactly the news feed pattern: take the top few hundred globally-ranked stories (plus the user's followed topics), apply the personalization boost, take the top 20. Major breaking news should still surface even if off-interest — so blend a personalization boost into the global score rather than filtering to only-interests (a pure filter creates a filter bubble that hides a national emergency from someone who only reads sports).

Serving: cache-friendly and fresh. The feed page is served from the ranked story cache. Because the global top stories are the same for everyone (before personalization), the global ranked list and each story's card are highly cacheable — and a breaking story is a read hot key (millions open it at once), so its cache entry is replicated across nodes so no single shard melts, exactly the celebrity/hot-key handling from the news feed. Section/topic pages are just pre-filtered ranked lists.

Freshness vs stability. News ranking constantly changes as coverage grows — but a feed that reshuffles while you're reading it is disorienting. Resolve it the same way the news feed does: cursor pagination serves a stable page (items don't shuffle under you within a session), while freshness means new/updated stories are ready on the next load or a "new stories" refresh — not live-injected into the page you're scrolling.

Tradeoffs & bottlenecks

• Exact-dup vs semantic clustering (cost vs recall). Fingerprint dedup is cheap and precise but only catches reprints; embedding-based clustering catches differently-worded coverage but costs an embedding + vector index per new article. The design runs fingerprints first (cheap, high volume) and embeddings only on survivors — buying semantic recall at bounded cost.
• Precompute clusters vs cluster at query time. Clustering is expensive and shared across all readers, so it's precomputed on ingest and materialized as Story records. Query-time clustering would repeat that work per request and blow the read latency budget — never do it.
• Freshness vs stability of the feed. Aggressive re-ranking keeps the feed current but reshuffles it under the reader. Cursor pagination + "new stories" refresh gives freshness (ready immediately) without live reshuffling (stable within a session).
• Global vs personalized ranking. A single global ranking is cheap and cacheable but identical for everyone; full per-user ranking is tailored but uncacheable and expensive. The compromise — precomputed global score + cheap read-time personalization boost over a candidate set — gets most of the benefit of both.
• Clustering precision vs recall. Too-eager merging fuses distinct events into one story (very visible error); too-timid leaves one event split across stories (less visible). Favor precision; tune thresholds toward not-merging when unsure.
• Freshness vs crawl cost. Polling everything every minute wastes budget; adaptive per-source polling spends it where news appears, at the cost of per-source scheduling state and an occasional poll-late on a source that suddenly breaks a story.

Extensions if asked

Add only the extension that changes the design discussion:

• Topic / section pages. Classify each story into topics (politics, sports, tech) at ingest; a section page is a pre-filtered, pre-ranked story list. Cheap once stories carry topic[].
• Trending / breaking detection. Watch cluster velocity — the rate of new sources joining — to flag a breaking story before its absolute source_count is large, and surface it prominently (or push a notification).
• Multi-language + translation. Detect each article's language at ingest, cluster within-language (or across, via multilingual embeddings so the same event in English and Spanish can merge), and optionally translate the canonical headline for the reader's locale.
• Misinformation / quality signals. Fold source-reliability and fact-check signals into avg_source_authority so low-quality sources are down-weighted in ranking and can't hijack a story's canonical pick. A hard filter for known-bad sources; a soft down-weight otherwise.
• Breaking-news notifications. For high-velocity, high-importance clusters matching a user's interests, push a notification — a fan-out through a notification queue, reusing the news feed's notification path.

What interviewers look for & common mistakes

What interviewers usually reward:

• Making clustering the centerpiece — recognizing the product's job is to collapse many articles about one event into a single story, not to list articles.
• Two-layer dedup/clustering — cheap fingerprints (hash + SimHash) for reprints, embeddings for differently-worded same-event coverage — done incrementally over a fingerprint/vector index, never a scan.
• News-scale freshness — minutes not days, via RSS/sitemap discovery and adaptive per-source polling, explicitly reusing the web crawler for politeness/frontier.
• Read-heavy serving — precompute the global story ranking, personalize cheaply over a candidate set at read time, cursor pagination, hot-story replication — reusing the news feed.
• Using source_count (coverage volume) and source authority as first-class ranking signals — signals that only clustering can produce.

Before you finish, do a quick mistake check:

• Did you make the feed return stories (clustered), not a flat time-sorted list of near-identical articles?
• Did you dedup and cluster — fingerprints for reprints and embeddings for same-event coverage — and do it incrementally via an index, not a scan?
• Did you crawl for minutes-scale freshness (RSS/sitemaps, adaptive polling), reusing the web crawler's politeness/frontier instead of re-deriving it?
• Did you rank by freshness + importance (coverage volume, source authority), not raw publish time?
• Did you precompute global ranking and personalize over a small candidate set at read time (not per-user over the whole corpus)?
• Did you pick a canonical article per cluster and cap/replicate hot stories on the serving path?