Design a Price Tracker

CamelCamelCamel Price Alert System Design

Overview

A price tracker watches product prices across retailers and notifies users when a price drops below a threshold they set — "tell me when this monitor falls under $250." CamelCamelCamel does this for Amazon; Honey, Keepa, and many deal sites do similar work. The user-facing feature is simple, but the system has three hard problems: acquiring prices at scale (crawling/polling millions of products without getting blocked or doing duplicate work), storing price history as a time series, and evaluating millions of user alert thresholds efficiently so a price change does not scan every alert in the system.

The naive design — "crawl everything every minute, and on each price change re-check all alerts" — does far too much work. Strong answers show smart scheduling and dedup for crawling, a time-series store for history, and an indexed, threshold-aware alert evaluation that touches only the alerts that could possibly fire, plus a notification fan-out that survives bursts.

In this walkthrough you'll scope it, size it, model the data, design the API, draw the ingest, alert, and read paths, and deep-dive crawling at scale, history storage, efficient alert evaluation, and reliable notification delivery.

Functional requirements

• Track a product. A user adds a product (by URL or id) to watch.
• Set an alert. A user sets a threshold (e.g. "notify me under $X", or "notify on any drop ≥ Y%").
• Acquire prices. The system periodically fetches current prices for tracked products.
• Store and show price history. Users can view a product's price over time (the chart).
• Notify. When a product's price meets a user's condition, send a notification (email/push).

Out of scope (state it): the retailers' own systems, payment/checkout, and a full deals-discovery feed. We keep history and alert evaluation central — they drive the storage and the hard algorithmic part.

Non-functional requirements

• Scalable ingestion. Track tens of millions of products; refresh them at appropriate cadences without hammering retailers or duplicating work.
• Efficient alert evaluation. A price change must trigger only the alerts it could satisfy — never a scan of all alerts. This is the most important non-functional requirement.
• Timely but not real-time. Price drops can notify within minutes; this is not a trading system. Eventual consistency and minute-scale latency are fine.
• Polite crawling / resilience. Respect rate limits and robots, survive blocks and layout changes, dedup work, and retry transient failures.
• Durable history. Price history is the product's value; never silently lose it.

Estimations

State assumptions; the aim is to justify the ingestion and alert designs.

Rounded assumptions

• Tracked products: 50M.
• Average refresh cadence: every 6 hours (4×/day); hot/popular products more often, long-tail less.
• Users: 10M; average 5 active alerts each → 50M alerts.
• Crawling all products at that cadence means ~200M fetches/day — about ~2.3K fetches/sec average (call it ~2–5K/sec with retries and peak headroom).
• If about 10% of crawls find a price change, expect about ~20M price changes/day, or roughly ~200–300 changes/sec.
• History volume: ~20M changes/day × 365 ≈ 7B points/year, at ~50 bytes each ≈ low-single-digit TB/year of raw history. Modest per year, but it is append-only and grows forever — which is why a time-series store with downsampling/retention beats one ever-growing relational table.

How the numbers affect the design

Core entities / data model

• Products live in a relational/KV store; next_crawl_at drives scheduling.
• PricePoint lives in a time-series database (TimescaleDB, or a wide-column store like Cassandra) partitioned by product_id and ordered by ts — append-heavy writes, range reads for the chart.
• Alert lives in a relational store with a critical composite index on (product_id, type, threshold) — this index is what makes efficient evaluation possible (deep dive 3).

API design

A small RESTful API covers the whole product — track a product, set alerts, and read price history. Let HTTP status codes carry the result.

Track a product

POST /api/products/track
Body: { "url": "https://retailer.com/item/123" }
201 Created
Returns: { "product_id": "..." }

Start watching a product from its URL. The server resolves the URL to a canonical product_id — deduping so two users watching the same product share one crawl — and schedules it for crawling.

Create an alert

POST /api/alerts
Body: { "product_id": "...", "type": "under", "threshold": 250 }
201 Created
Returns: { "alert_id": "..." }

Attach a price alert to a tracked product. type is under (absolute, "below $250") or pct_drop (relative); threshold is the trigger value. The alert is stored indexed by (product_id, type, threshold) — the index that makes efficient evaluation possible (deep dive 3).

Price history (chart)

GET /api/products/{id}/history?range=90d
200 OK
Returns: { "points": [ { "ts": ..., "price": ... }, ... ] }   (cached)

Fetch the price-history time series for the chart. Reads are cached and downsampled to the requested range, so the endpoint never scans the full raw series for popular products.

Delete an alert

DELETE /api/alerts/{id}
204 No Content

Stop and remove an alert. 204 No Content is the standard "deleted, nothing to return" response.

There is deliberately no "send notification" endpoint: notifications are pushed by the system (crawl detects a change → evaluator matches alerts → notification queue), never pulled by the client.

High-level architecture

Three flows make up the system: an ingest path that crawls prices, an alert path that turns a price change into a notification, and a read path that serves the price chart. We'll walk each one on its own, then combine them.

1. Ingest path

The scheduler decides what to crawl and when; workers fetch prices and write a new history point only when the price actually changed.

flowchart LR
    Sched[Scheduler] -->|"products where next_crawl_at <= now"| CrawlQ[[Crawl Queue]]
    CrawlQ --> Workers[Crawl Workers]
    Workers -->|"fetch + parse"| Retailers[(Retailer Sites)]
    Workers -->|"on change: write point"| TSDB[(Price History TSDB)]
    Workers -->|"update current_price + next_crawl_at"| ProdDB[(Product DB)]
    Workers -->|"on change: emit event (old + new price)"| ChangeQ[[Price-Change Queue]]

1. The scheduler finds products whose next_crawl_at is due and places them on the crawl queue.
2. Crawl workers fetch product pages from retailers and parse the current price.
3. If the price did not change, the worker just updates crawl metadata (next_crawl_at) and stops — no history write, no event.
4. If the price changed, the worker writes a new PricePoint, updates current_price, and emits a price-change event.

The change event is the hinge: it is the only thing that wakes the alert path, so the ~90% of crawls that find no change cost nothing downstream.

2. Alert path

A price-change event is matched against the alerts on that product, and matches become notifications.

flowchart LR
    ChangeQ[[Price-Change Queue]] --> AlertEval[Alert Evaluator]
    AlertDB[(Alert DB - indexed product_id, threshold)] --> AlertEval
    AlertEval -->|"matched alerts"| NotifQ[[Notification Queue]]
    NotifQ --> Notif[Notification Service]
    Notif --> Users[Email / Push]

1. The alert evaluator consumes a price-change event for some product P.
2. Using the (product_id, type, threshold) index, it selects only the alerts on P that the new price now satisfies — not the whole alert table (deep dive 3).
3. Matched alerts are enqueued on the notification queue.
4. The notification service delivers email/push with retries and per-user batching, decoupled from evaluation so a slow provider can't back up the crawl.

3. Read path

The price chart is a pure read, served from cache.

flowchart LR
    Client[User] -->|"view chart (range=90d)"| Cache[(History Cache)]
    Cache -->|"hit: points"| Client
    Cache -.->|"miss"| TSDB[(Price History TSDB)]
    TSDB -.->|"downsampled range"| Cache

1. The user opens a product's price chart.
2. On a cache hit, the cached, downsampled series is returned immediately.
3. On a miss, the service range-scans the time-series store for the requested range, writes the result back to cache, and returns it. Charts are read far more than written, so this keeps the TSDB off the hot path.

4. Combined architecture

Putting the three together: crawling feeds both history and the change events that drive alerts, while chart reads stay on their own cached path.

flowchart LR
    Sched[Scheduler] -->|"due products"| CrawlQ[[Crawl Queue]]
    CrawlQ --> Workers[Crawl Workers]
    Workers -->|"fetch + parse"| Retailers[(Retailer Sites)]
    Workers -->|"on change: write point"| TSDB[(Price History TSDB)]
    Workers -->|"update current_price"| ProdDB[(Product DB)]
    Workers -->|"on change: emit event (old + new price)"| ChangeQ[[Price-Change Queue]]
    ChangeQ --> AlertEval[Alert Evaluator]
    AlertDB[(Alert DB - indexed product_id, threshold)] --> AlertEval
    AlertEval -->|"matched alerts"| NotifQ[[Notification Queue]]
    NotifQ --> Notif[Notification Service]
    Notif --> Users[Email / Push]
    Client[User] -->|"view chart"| Cache[(History Cache)]
    Cache -.->|"miss"| TSDB

Queues at each boundary (crawl, change, notification) keep the stages decoupled, so a burst or a slow downstream in one stage does not stall the others.

Deep dives

1. Crawling / polling at scale with scheduling and dedup

Refreshing 50M products at ~2.3K fetches/sec needs more than a loop. The design is a scheduler + distributed crawl workers + a queue:

• Scheduling by next_crawl_at. Each product carries a next_crawl_at; the scheduler periodically enqueues those that are due, then bumps next_crawl_at forward by the product's cadence. Seed next_crawl_at with random jitter (and re-jitter on each bump) so due-times spread evenly rather than clustering — e.g. after a bulk import that would otherwise make everything due at once. This spreads fetches evenly instead of a thundering daily batch.
• Adaptive cadence. Not every product deserves the same frequency. Crawl hot / volatile / many-watchers products often (minutes–hours) and stale / never-changing products rarely (days). This concentrates the fetch budget where price changes actually happen and respects retailer load.
• Dedup. Two users tracking the same product must not cause two crawls — track by canonical product_id, so one fetch serves all watchers. Also dedup in-flight work: a distributed lock or a "crawling" flag prevents two workers fetching the same product at once.
• Politeness & resilience. Respect robots/rate limits per retailer (token-bucket per domain), rotate IPs/user-agents where permitted, back off on 429/blocks, and isolate parsers per retailer so a layout change breaks one parser, not the fleet. Retry transient failures with exponential backoff; quarantine products that fail repeatedly.
• Change-only writes. Compare the fetched price to current_price; only on change, write a PricePoint and emit an event carrying (product_id, old_price, new_price) — the old price lets the evaluator judge percent-drop alerts without a re-read. Since only ~10% of polls change the price (see estimations), this keeps history storage roughly ~10× smaller and avoids firing the alert path on no-ops.
• Guard against the read-modify-write race. Two workers (or a retry racing the original) can both read the old current_price, both see a change, and both write a PricePoint + emit — duplicate history and a double fire; make the update a compare-and-set (UPDATE ... SET current_price = new WHERE product_id = P AND current_price = old) so only one writer wins, and let the change event carry the deterministic change_event_id from deep dive 4 so any duplicate that slips through dedups.

2. Storing price history as a time series

Price history is append-heavy writes and range-scan reads ("last 90 days") — a textbook time-series workload.

• Store one point per change, not per poll — (product_id, ts, price). Most polls find no change, so this slashes volume (see estimations).
• Partition by product_id, order by ts. A chart query is a single-partition range scan — fast. A time-series DB (TimescaleDB hypertables, or Cassandra/wide-column) is built for this; a naive single relational table of billions of rows would scan poorly.
• Downsample old data. Recent history at full resolution; older history rolled up (daily min/max/close) to shrink storage and speed long-range charts. Retention policies drop or archive very old raw points.
• Cache chart responses for popular products keyed by (product_id, range); invalidate on a new point. The chart is read far more than it's written.

3. Efficient alert evaluation — don't scan every alert

This is the subtle, high-signal part. Naive evaluation re-checks all 50M alerts on every price change — that's ~20M changes/day × 50M alerts ≈ 10¹⁵ (a quadrillion) comparisons/day (see estimations). The fix is to make evaluation proportional to the alerts on the changed product, not the whole table.

• Index alerts by product. Alerts are stored with a composite index on (product_id, type, threshold) — type sits before threshold because that column holds a price for under but a percent for pct_drop, so a range-prune on threshold is only meaningful within a single type. A price change for product P selects only WHERE product_id = P — the handful of alerts on P, not all 50M.
• Threshold-prune within the product. Don't even check every alert on P; the index on threshold lets you fetch only alerts whose condition the new price now satisfies — for an under alert, WHERE product_id = P AND type = 'under' AND threshold >= new_price returns the alerts now satisfied — then edge-trigger (using last_fired_at / comparing against old_price) to fire only those that were NOT already satisfied at old_price. pct_drop alerts can't use that range-prune (their threshold is a percentage, not a price, sitting in the same column — a percent 20 would sort among prices): compute the drop of new_price against each alert's reference_price and fire those whose drop meets threshold. That still costs only O(alerts on P) — the product's handful, not the whole table.
• Edge-trigger, don't level-trigger. Fire when the price crosses into the satisfying range, not on every subsequent poll while it stays low — otherwise a price that sits below $250 spams the user every cycle. Track last_fired_at (or the price at last fire) and only re-fire on a new crossing or after a cooldown. For pct_drop, how you handle reference_price after a fire is a product decision, not one correct behavior: ratchet from last fire (reset reference_price to the fired price, so each subsequent drop is measured fresh) catches a stair-step decline without re-alerting on the same drop, while fixed baseline with cooldown (keep the original alert-set reference_price and suppress re-fires for a window) keeps the alert anchored to the price the user first saw. Both are legitimate — pick per how you want repeated drops to read to the user; just don't leave it unspecified, or a slow decline will either fire once and never again or spam on every dip past a fixed baseline.
• Event-driven, not polling the alerts. The evaluator reacts to change events from the crawl path (the (product_id, threshold) lookup), so work happens only when a price actually moved — ~200–300 price-change events/sec (see estimations), each touching just that product's alerts, not 10¹⁵ comparisons/day.

4. Notification fan-out

Matched alerts go onto a notification queue drained by notification workers, decoupled from evaluation so a slow provider never back-pressures the crawl/alert path. Three things matter:

• Batch, don't spam. If a user's whole wishlist goes on sale, send one digest, not 20 messages — coalesce a user's alerts within a short window, and respect per-user rate limits and quiet hours. (A viral product can fire thousands of alerts at once; collapse the duplicated work — deep dive 5.)
• Idempotent delivery (queues are at-least-once). Notification queues are usually at-least-once, and provider retries add more duplicates — so make sending idempotent: dedup on (alert_id, change_event_id), or reuse the last_fired_at guard from deep dive 3, so retries and re-crossings don't double-send. For this to hold, change_event_id must be deterministic — derived from the crossing itself (e.g. hash of product_id + new_price + observed_ts, or the PricePoint primary key), not a fresh UUID per emit; otherwise a re-crawl that re-detects the same crossing produces a new id and the dedup silently fails, so lean on last_fired_at as the real guard when in doubt.
• Channels. Deliver over the channels the user opted into (email / push / SMS). Reliable delivery at this scale — handling bounces, dead push tokens, unsubscribes, and protecting sender reputation so providers don't throttle you — is a notification-system design problem of its own, beyond this guide (see Extensions).

5. Backpressure: handling a full queue

Queues smooth bursts but aren't infinite. First classify the backlog:

• Transient burst — drains once it passes; the queue is doing its job, so let it drain.
• Sustained overload — producers consistently outpace consumers; no buffer fixes this. Add consumer capacity or cut the load.

Responses, in order: scale consumers (autoscale on queue depth — the signal to alert on); backpressure the producer (here the crawler self-limits via next_crawl_at — fall behind and it stops enqueuing and stretches cadence); coalesce / batch to cut volume (only the latest price per product matters); and load-shed the least valuable last (a price-change event older than the current price is stale, safe to drop). A dead-letter queue keeps poison messages from clogging retries.

The one-line version: a queue converts a throughput problem into a latency problem — it smooths bursts but can't cure a sustained deficit.

Tradeoffs & bottlenecks

• Freshness vs crawl cost. Crawling everything frequently gives fresher prices but costs fetches and risks blocks. Adaptive cadence trades freshness on cold products for budget on hot ones.
• Storage vs resolution. Storing every poll gives perfect fidelity but ~10× the data (only ~10% of polls actually change the price); storing on-change + downsampling old data trades some fidelity for far less storage. Change-only writes are the standard call.
• Index maintenance vs query speed. The (product_id, type, threshold) index makes evaluation cheap but adds write cost when alerts change — acceptable since alerts change far less often than they're evaluated.
• Edge vs level triggering. Edge-trigger avoids spam but needs state (last_fired_at); getting this wrong is a top user complaint (repeated notifications).
• Crawl fragility. Per-retailer parsers break on layout changes — isolate them and monitor parse-success rate; a silent parse failure looks like "price never changes."
• Politeness vs throughput. Aggressive crawling gets you blocked; per-domain rate limiting protects access at the cost of slower refresh for big retailers.
• Queue backlog under bursts. A site-wide sale spikes both price changes and notifications; queues absorb the burst but latency rises under load. Autoscale consumers on queue depth, backpressure the crawler, and coalesce/batch to cut volume (deep dive 5).

Extensions if asked

If the interviewer wants to push beyond the core price tracker, add only the extension that changes the design discussion:

• Rate limits and anti-blocking. Add per-retailer token buckets, backoff, and parser isolation.
• Feed of popular deals. Turn price changes into a ranked deal feed.
• Checkout or inventory accuracy. If users can buy through the platform, price tracking becomes commerce.
• Notification delivery at scale. Multi-channel delivery, batching/digests, handling bounces and dead push tokens, honoring unsubscribes, and protecting sender reputation are a system in their own right — the part this guide deliberately keeps shallow. There's no dedicated guide yet.

What interviewers look for & common mistakes

What interviewers usually reward:

• Indexed, threshold-aware alert evaluation — recognizing the naive all-alerts scan costs changes × total alerts (≈10¹⁵/day) and fixing it with (product_id, type, threshold) so each change touches only the alerts that can fire. This is the differentiator.
• Scheduler + distributed crawlers + queue with next_crawl_at, adaptive cadence, and dedup by canonical product.
• Time-series storage partitioned by product, change-only writes, and downsampling — with the math showing why.
• Edge-triggering with last_fired_at to avoid notification spam.
• Decoupled notification fan-out via a queue with batching and retries.
• Backpressure and load-shedding when queues fill — autoscaling on depth, slowing producers, and coalescing or dropping stale events instead of letting latency grow unbounded.

Before you finish, do a quick mistake check:

• Did you avoid scanning all alerts on every price change?
• Did you use scheduling, queues, dedup, and adaptive cadence for crawling?
• Did you store price history on change instead of every poll?
• Did you partition or downsample time-series history?
• Did you edge-trigger alerts to avoid notification spam?
• Did you decouple notification delivery from alert evaluation?
• Did you handle queue overload with backpressure and scaling, not an unbounded buffer?