Design a Metrics Monitoring & Alerting System

Datadog / Prometheus System Design

Overview

Every production service emits a stream of numbers: CPU usage, request latency, error rates, queue depths. A metrics monitoring and alerting system — the kind Datadog, Prometheus, and InfluxDB provide — collects those numbers from hundreds or thousands of hosts, stores them in a time-series database, makes them queryable for dashboards, and evaluates alert rules that page an on-call engineer when something breaks. The product sounds like "store some numbers and compare them to a threshold," but at the scale of thousands of hosts, hundreds of metrics each, and sub-minute resolution it hides hard problems: you cannot keep raw data forever (storage explodes), querying raw high-resolution data over days is slow, and an alerting pipeline that silently stops evaluating rules defeats its own purpose.

It is labeled "Medium" because the naive design breaks in specific, teachable ways. If you store raw time-series at full resolution forever, storage compounds weekly and dashboards scanning months of fine-grained data are impossibly slow. If you store one series per unique combination of metric name and tag values without bounding that space, cardinality explosion makes the TSDB unmanageable. And if the alerting engine evaluates rules against a single datastore with no deduplication, one outage fires hundreds of duplicate pages and fatigues your on-call team.

A strong answer downsamples older data into coarser rollups with clear retention tiers, enforces tag cardinality limits at ingest time, serves dashboard queries from pre-aggregated data, and builds an alerting pipeline with deduplication, grouping, and self-monitoring so silence means "everything is fine" rather than "the alerting system crashed."

This system differs from an ad click aggregator, which counts a single event type into per-(id, minute) buckets. Here you are collecting many metrics from many sources, storing them in a time-series database with rollups and retention tiers, and adding an entirely separate alerting pipeline — the combination of TSDB design, cardinality management, and reliable alert delivery is what makes this question distinct.

Out of scope (say so): distributed tracing and log aggregation (though we note where they correlate), the agent SDKs or client libraries used by services to emit metrics, and the SaaS billing model for the monitoring product itself.

Functional requirements

• Ingest metrics. Accept timestamped metric data points from thousands of services and hosts — each data point carries a metric name, a value, a timestamp, and a set of tags/labels (e.g. host=web-01, region=us-east, service=payments).
• Query metrics. Return a time series or aggregated value (sum, avg, percentile) for a metric over a time range, filtered and grouped by tags. Power dashboards and ad-hoc analysis.
• Dashboards. Render near-real-time charts (last hour, last day, last week) with low latency. Charts must update within seconds of new data arriving.
• Alert rules. Let operators define rules like "alert if p99_latency{service=payments} exceeds 500ms for 5 consecutive minutes." Evaluate rules continuously and fire notifications (email, PagerDuty, Slack) when conditions are met.
• Alert management. Silence an alert during a known maintenance window; group related alerts (same service, same region) into one page.

Out of scope (state it): log ingestion and distributed tracing (separate systems, though we mention correlation), the client-side SDK used to emit metrics, and the on-call scheduling system that routes pages to specific engineers.

Non-functional requirements

• Massive write volume. Thousands of hosts × hundreds of metrics × one data point every 10–60 seconds = tens of millions of data points per minute. The ingest path must absorb bursts without dropping data.
• Low-latency dashboard reads. A chart over the last hour should render in under a second; a week-long chart in a few seconds. Reads scan a large time range but must feel fast — which requires pre-aggregated data, not raw-scan-every-time.
• Long-term storage with controlled cost. Keeping full resolution forever is prohibitive; data must be downsampled into coarser rollups as it ages, and eventually purged after a retention period.
• Bounded cardinality. Each unique combination of (metric name + tag set) creates a separate series. Unbounded high-cardinality tags (like user_id) can create billions of series and crash the TSDB. The system must enforce cardinality limits.
• Reliable alerting. A missed alert during an outage is worse than a false positive. The alert evaluation pipeline must be highly available, and its own health must be monitored.
• Eventual consistency is fine for dashboards. A chart that is a few seconds stale is acceptable; we need correctness over time windows, not sub-second freshness.

Estimations

State assumptions; the goal is to justify the storage model and the rollup design, not to be exact.

Rounded assumptions

• Hosts: ~10,000. (A mid-to-large fleet; a Datadog-scale system would be millions, but 10K is a reasonable interview scope.)
• Metrics per host: ~200 (CPU, memory, disk, network, and dozens of application-level metrics).
• Scrape interval: ~15 seconds (a common high-resolution config; Prometheus defaults to 1m — we use 15s here as a stated assumption for the math).
• Raw data points/sec: 10,000 hosts × 200 metrics ÷ 15s ≈ ~130,000 data points/sec.
• Raw storage per point: ~16 bytes before compression (8-byte timestamp + 8-byte float value). Metric name and tag metadata are stored once per series/chunk, not repeated on every point. At full resolution that is ~16 bytes × 130K/s × 86,400 s/day ≈ ~180 GB/day raw values before compression and index overhead — manageable short-term but ~65 TB/year at full resolution.
• With rollups + retention: keep raw (15s resolution) for 7 days, 1-minute rollups for 30 days, 1-hour rollups for 1 year. Storage compresses by ~50–100× — the year-long store is hundreds of GB rather than tens of TB.
• Series count (cardinality): 10,000 hosts × 200 metrics × (say) a few tag-value variants each = ~5–10M active series at any moment. Adding user_id to even ONE metric alone creates 100M series for that metric — which is why cardinality is a first-class constraint. (If all 200 metrics carried user_id, the combinatorial total would reach 20 billion series, but a single offending metric is enough to crash the TSDB.)
• Alert rules: ~10,000 (one system with many teams). Each rule evaluates roughly every minute: 10,000 rules × one TSDB range query/min = ~170 query/sec from the alert engine alone.

How the numbers affect the design

Core entities / data model

Three things are stored: the metric series definition (what series exists, its metadata), the raw data points (the source of truth), and the rollup/aggregate data (derived, used for long-range queries and alerting). Alert rules and notification channels live in a relational config store.

The cardinality problem lives in Series. A series is uniquely identified by its metric name plus its full tag set — every unique combination of tag values for a metric is a separate series. Consider http_requests_total{method, status_code, handler, user_id}: if user_id has 100M distinct values and appears in the tag set, you have 100M separate series for this one metric. TSDBs hold a large in-memory inverted index mapping tag values to series ids for fast filter/group queries — that index fits in memory only if series count is bounded. The fix: prohibit high-cardinality dimensions as tags; encode them in log events or traces instead, which are built for per-request detail.

The RollupBucket stores min, max, sum, and count — not just the average. Storing all four lets you compute the average (sum/count) and rebuild accurate re-rollups: rolling up hourly buckets into daily buckets means summing their sum and count fields, not averaging their averages (which would give wrong results when bucket sizes differ).

For percentile metrics such as latency p99, scalar min/max/sum/count is not enough. Store a histogram or sketch (for example Prometheus-style buckets or DDSketch) alongside the rollup for latency series. Percentiles are computed by merging histograms, not by averaging p99 values.

API design

Four operations carry the system: write data points, query a time series, manage alert rules, and query current alert state.

Write (ingest) data points

POST /api/v1/metrics/write
Body: {
  "series": [
    {
      "metric": "cpu_usage_percent",
      "tags": { "host": "web-01", "region": "us-east" },
      "points": [
        { "timestamp": 1751760000, "value": 72.4 },
        { "timestamp": 1751760015, "value": 73.1 }
      ]
    },
    …
  ]
}
204 No Content

Batched write. The client sends many series and many points per series in one request — not one HTTP call per data point. The endpoint validates metric names and tags, enforces cardinality limits (reject series that would push a metric over its max series count), and enqueues the batch for durable ingestion. Returns 204 (no body) because there is nothing to return; ingest is asynchronous.

Query a time series

GET /api/v1/query_range
  ?metric=cpu_usage_percent
  &tags=host:web-01,region:us-east
  &from=2026-07-05T00:00:00Z
  &to=2026-07-06T00:00:00Z
  &step=1h
  &fn=avg
200 OK
Returns: {
  "metric": "cpu_usage_percent",
  "tags": { "host": "web-01", "region": "us-east" },
  "step": "1h",
  "series": [
    { "t": 1751673600, "v": 61.3 },
    { "t": 1751677200, "v": 58.8 },
    …
  ]
}

Range query with aggregation. step specifies the resolution of the returned series; the backend selects the appropriate rollup tier (raw for sub-minute steps, 1m rollups for hour-long ranges, 1h rollups for multi-day ranges) to keep response time low. fn is the aggregation function applied within each step bucket: avg, sum, min, max, or a percentile. Percentiles require histogram/sketch-backed series; scalar rollups can answer avg, sum, min, and max only. When tags contains a wildcard or a grouping directive (e.g. host=* + group_by=region), the backend aggregates across multiple series — which is the expensive path that rollups and caching serve.

Alert rule CRUD

POST /api/v1/alert_rules
Body: {
  "name": "payments-latency-high",
  "query": "p99(http_request_latency_ms{service=payments})",
  "condition": "gt",
  "threshold": 500,
  "duration": "5m",
  "notification_channels": ["pagerduty-primary"]
}
201 Created
Returns: { "rule_id": "rule_abc123", … }

GET    /api/v1/alert_rules/{rule_id}
PUT    /api/v1/alert_rules/{rule_id}
DELETE /api/v1/alert_rules/{rule_id}

Create, update, and delete alert rules. The duration field means "the condition must hold continuously for this long before firing" — this prevents transient spikes from paging your on-call team. On creation the system validates that the query is syntactically valid and references a known metric. A p99 alert is valid only for metrics stored as histograms/sketches; a threshold on avg or max can use plain scalar rollup buckets.

Query alert state

GET /api/v1/alerts?state=firing
200 OK
Returns: {
  "alerts": [
    {
      "rule_id": "rule_abc123",
      "name": "payments-latency-high",
      "state": "FIRING",
      "fired_at": "2026-07-06T02:14:00Z",
      "value": 612.5,
      "labels": { "service": "payments" }
    },
    …
  ]
}

Returns currently active (or recently resolved) alerts. Used by the on-call dashboard to see what is firing right now.

High-level architecture

Three paths carry the system: an ingest / write path (metrics flow from hosts into the TSDB), a read / query path (dashboards query stored data), and an alerting path (rules are evaluated periodically and notifications fired). We'll walk each, then combine them.

1. Ingest / write path

Metric data flows from agents on hosts, through a durable buffer, into the time-series store.

flowchart LR
    Agents["Metric agents<br/>(on each host)"] -->|"POST /write<br/>(batched)"| GW["Ingest Service"]
    GW -->|"validate + enqueue"| Q[["Ingest queue<br/>(partitioned stream)"]]
    Q --> W["TSDB Write Workers<br/>(batch ingest)"]
    W -->|"append raw data points"| TSDB[("Time-series DB<br/>(raw, high-res)")]
    W -->|"register new series"| Meta[("Series metadata<br/>+ inverted index")]
    TSDB -.->|"background rollup job"| Roll["Rollup engine"]
    Roll -.->|"wait late-arrival window, then seal"| Roll
    Roll -->|"1m + 1h summaries"| TSDB

1. Agents on each host collect metrics and push a batch to the Ingest Service every 15–60 seconds (the push model). Batching is critical: 130K data points/sec means one-at-a-time inserts would create unmanageable write amplification.
2. The Ingest Service validates tag names, enforces cardinality limits (per-metric series count cap), and enqueues the batch into a partitioned ingest queue (e.g. Kafka, partitioned by metric name so a metric's data lands on one partition and can be ordered). The queue is the buffer that absorbs ingest bursts without stalling the TSDB.
3. TSDB Write Workers consume from the queue in batches and perform bulk appends to the time-series database. They also register new series in the series metadata store and update the in-memory inverted index (tag → series_id map) that filter queries rely on.
4. A background Rollup Engine scans raw data on a cadence, computes min/max/sum/count rollup buckets at 1-minute and 1-hour resolution, and writes them back into the TSDB. Before sealing a rollup bucket, the engine waits a configurable late-arrival window (e.g. 2–5 minutes past the bucket boundary) to absorb out-of-order or retried data points before treating the bucket as final. It also enforces retention: delete raw data older than 7 days, 1-minute rollups older than 30 days, and 1-hour rollups older than 1 year.

2. Read / query path

Dashboard queries are served from the appropriate rollup tier, never by scanning the full raw data.

flowchart LR
    Dash["Dashboard<br/>(user)"] -->|"GET /query_range"| QS["Query Service"]
    QS -->|"resolve series_ids<br/>via inverted index"| Meta[("Series metadata<br/>+ inverted index")]
    QS -->|"select rollup tier<br/>raw / 1m / 1h"| TSDB[("Time-series DB")]
    TSDB -->|"range scan → buckets"| QS
    QS -->|"aggregate across series"| QS
    QS -.->|"cache result"| Cache[("Query cache<br/>(short TTL)")]
    QS -->|"time-series response"| Dash

1. The Query Service receives a range query and first resolves the matching series ids by looking up the tag filter in the inverted index (e.g. "all series where service=payments and metric=http_request_latency_ms").
2. It selects the right rollup tier based on the requested time range and step size: raw 15s data for the last few minutes, 1m rollups for multi-hour ranges, 1h rollups for multi-day ranges. This is what keeps dashboard reads fast — a week-long chart scans at most 168 hourly buckets per series instead of ~40,000 raw points.
3. It performs a range scan in the TSDB (which stores data sorted by (series_id, timestamp) so a time range is a contiguous read) and aggregates the results across the requested step buckets and across multiple series if a group-by was specified.
4. Results are cached with a short TTL (typically 15–60 seconds, matching the scrape interval). A dashboard refreshing every 30 seconds should usually hit the cache.

3. Alerting path

Alert rules are evaluated periodically; notifications are deduplicated, grouped, and delivered reliably.

flowchart LR
    Rules[("Alert rules")] --> Eval["Rule Evaluation Engine<br/>(periodic, sharded)"]
    TSDB[("Time-series DB")] -->|"recent window query"| Eval
    Eval -->|"write state"| AS[("Alert State Store")]
    Eval -->|"on FIRING transition"| Dedup["Dedup &amp; Grouping Service"]
    Dedup -->|"grouped notification"| Notif["Notification Dispatcher"]
    Notif -->|"PagerDuty / email / Slack"| OnCall["On-call engineer"]
    Notif -.->|"retry on failure"| Notif

1. The Rule Evaluation Engine queries the TSDB for the last N minutes of data for each rule's metric selector, evaluates the condition, and tracks a state machine (OK → PENDING → FIRING → RESOLVED) for each (rule, label-set) tuple.
2. A rule transitions to FIRING only after its condition holds for the full duration window (e.g. 5 consecutive minutes above threshold). This two-stage PENDING → FIRING transition is what prevents transient spikes from firing pages.
3. New FIRING transitions flow to the Dedup and Grouping Service, which suppresses duplicate notifications (the same rule firing repeatedly while already paged), groups related alerts (same service, same region) into a single page, and applies silences (maintenance windows).
4. The Notification Dispatcher delivers the grouped alert to the configured channels with at-least-once delivery — it retries on failure with exponential backoff and records delivery receipts. If PagerDuty is down, it retries; if it stays down, it falls back to email.

4. Combined architecture

flowchart LR
    Agents["Host agents"] -->|"push batches"| GW["Ingest Service"]
    GW --> Q[["Ingest queue"]]
    Q --> W["Write workers"]
    W --> TSDB[("TSDB<br/>(raw + rollups)")]
    W --> Meta[("Series metadata")]
    TSDB -.->|"rollup + retention"| Roll["Rollup engine"]
    Roll --> TSDB
    Dash["Dashboard"] --> QS["Query Service"]
    QS --> Meta
    QS --> TSDB
    QS -.-> Cache[("Query cache")]
    Rules[("Alert rules")] --> Eval["Rule eval engine"]
    TSDB --> Eval
    Eval -->|"write state"| AS[("Alert state")]
    Eval -->|"on FIRING transition"| Dedup["Dedup + grouping"]
    Dedup --> Notif["Notification dispatcher"]
    Notif --> OnCall["On-call"]

Ingest, query, and alerting are decoupled: a burst of ingest data hits the queue and write workers, never the query path. Dashboard reads serve rollups from the TSDB with a cache in front. The alert engine is a periodic reader with its own state store — a TSDB that is temporarily slow under write pressure still serves the alert engine's lightweight recent-window queries.

Deep dives

1. Ingestion and time-series storage — TSDB design, rollups, retention, and cardinality

This is the core of what makes a metrics system different from a general-purpose database. Three interlocking concerns: how data is physically stored for cheap appends and fast range scans, how rollups keep long-term storage tractable, and how cardinality is kept from spiraling.

TSDB storage model. Time-series data has a distinctive shape: for each series, values arrive in strictly increasing timestamp order, values are often similar to the previous one (making delta-compression effective), and reads always scan a contiguous time range per series. The natural physical layout is chunk-based and append-only (one compressed chunk per series per time window), sorted by (series_id, timestamp) — distinct from an analytical column store like Parquet, which organizes all values for one column across all rows:

• Timestamps are stored as delta-encoded integers (the delta from the previous timestamp in the chunk, not the full Unix timestamp) — a 15-second interval stored as 15 in 2 bytes rather than 1751760015 in 8 bytes, giving 4× compression.
• Float values are Gorilla-encoded (XOR of successive floats, then variable-length int), which compresses repeating or slowly-drifting values to 1–2 bytes per point.
• Data is organized into fixed-time chunks (e.g. two-hour blocks) per series; a chunk is sealed once its time window passes and is then immutable and compressible. Open (current) chunks live in memory; sealed chunks are compressed and flushed to disk or object storage.

This is essentially what Prometheus, InfluxDB, and the Facebook Gorilla paper describe. You do not need to invent it from scratch in an interview — just explain the shape and why it works: sequential disk I/O, cheap appends (writes always go to the open in-memory chunk), and fast range scans (a range query touches only the chunks whose time windows overlap the query range).

Rollups and retention tiers. The rollup engine runs as a background job, reading sealed raw chunks and writing pre-aggregated buckets:

• Raw (15s): keep for 7 days. Serves real-time dashboards and alert evaluation for recent data.
• 1-minute rollups: keep for 30 days. Serves dashboards up to a month back. Each 1-minute bucket stores min, max, sum, count of the 4 raw points in the minute.
• 1-hour rollups: keep for 1 year. Serves long-trend dashboards. Built by re-rolling the 1-minute buckets (sum the sum fields, sum the count fields — never average the averages).

After retention expires, the engine hard-deletes the corresponding chunks. Deletion does not need to be instant — a background sweep once per day is fine, since stale data that is past retention but not yet deleted is simply not returned by queries that respect the retention window.

The cardinality problem. The TSDB's inverted index maps every tag-value pair to the set of series that carry it. This index must fit in memory because filter queries (metric=cpu_usage AND host=web-01) use it on every read. At 5M active series with ~5 tags each, the index is ~hundreds of MB — manageable. Now add a tag user_id to even ONE metric: 100M users = 100M series for that single metric alone — the index no longer fits, query fanout is astronomical, and rollup jobs grind to a halt. (Adding user_id to all 200 metrics would reach 20B series, but one offending metric is already enough to bring the system down.)

Push vs pull for metric collection. The two canonical collection models:

• Pull (Prometheus-style): the monitoring system scrapes each target (host, service) on a schedule. Benefits: the monitoring system controls the scrape rate; a target that disappears from the scrape list is visibly absent (vs silently not sending). Drawback: the monitoring system must know about every target and reach it over the network — challenging in dynamic environments (auto-scaling, serverless, short-lived jobs).
• Push — buffered agent (Datadog-style): the agent on each host batches data points locally and flushes to the ingest service every 10–60 seconds. If the ingest service is briefly unavailable the agent buffers in memory and retries — no data loss for short partitions. Drawback: the monitoring system cannot distinguish "target is down" from "target chose not to send" — heartbeat metrics compensate for this.
• Push — fire-and-forget UDP (StatsD-style): the application sends individual counters/timers as UDP datagrams directly to a local StatsD daemon, which aggregates and forwards. UDP is zero-overhead for the caller but lossy under network congestion or daemon restart — suitable for high-frequency low-stakes counters, not for critical SLO metrics.

Most production systems support both: a push endpoint for agents and ephemeral workloads, and a pull scraper for services that expose a /metrics endpoint. For this design, push is the primary path.

2. Query path and dashboards — fast range reads, rollup selection, and caching

The query path is where rollups pay off. A correctly designed query service can serve a one-year trend chart in milliseconds by reading a few thousand hourly rollup buckets instead of hundreds of millions of raw points.

Rollup tier selection. The query service must automatically choose the coarsest tier that satisfies the requested step resolution:

For a 30-day chart at 1-hour resolution: 720 buckets per series — a trivially small scan. The same query at raw resolution: 172,800 points per series, 240× more data. Always use the coarsest tier that answers the question.

Fan-out across series for group-by queries. A query like "average CPU across all hosts in region=us-east" fans out to potentially thousands of series (one per host), reads their rollup buckets in parallel, and merges the results. The merge step is simple (sum the sum fields, sum the count fields, divide), but the fan-out cost scales with the number of matching series. Queries with wildcards over a high-cardinality tag (host=*) can match millions of series — which is another reason cardinality must be bounded. For large fan-outs, the query service reads in parallel across TSDB shards using the inverted index to identify which shards hold the relevant series.

Pre-aggregation for common dashboard queries. Dashboards for a given service always query the same few metrics with the same tag filters. Rather than re-reading rollup buckets on every dashboard refresh, a pre-aggregation job can periodically compute cross-series aggregates (e.g. "p99 latency across all hosts in service X, per minute") and store the result as a derived series. Dashboard queries then read one series instead of fanning out to thousands. This is optional depth — mention it as an optimization after explaining the base design.

Caching. Two layers:

1. Query result cache (Redis or in-process): cache the full result of a /query_range request with a TTL matching the scrape interval (~15–60s). A dashboard refreshing every 30 seconds hits the cache on the second and later fetches.
2. Series metadata cache (in-process): the inverted index (tag → series_id mapping) is used on every query to resolve matching series. At modest scale (5–10M series) each TSDB node can keep its full shard's index in memory — it's the TSDB's primary hot-path structure. At millions of series the index is sharded across TSDB nodes; each node keeps only the index for its own shard in memory, and the query service fans out to the relevant shards in parallel.

3. The alerting pipeline — rule evaluation, deduplication, grouping, and self-monitoring

The alerting pipeline is the part of a metrics system that most interviewers do not hear a strong answer on. It is not hard conceptually, but it has several failure modes that must be called out explicitly.

Rule evaluation. The rule engine is a periodic job (typically every 10–30 seconds per rule) that:

1. Executes the rule's query against the TSDB for the recent window (e.g. last 5 minutes).
2. Evaluates the condition (e.g. value > 500).
3. Advances the rule's state machine: if the condition is newly true, enter PENDING; if it has been true continuously for the duration threshold, transition to FIRING; if it becomes false, return to OK (resolved).

The PENDING → FIRING transition with a duration guard is critical. Without it, a single bad data point or a 10-second CPU spike fires a page. With it, only a sustained condition pages — reducing noise dramatically.

Scaling rule evaluation. At 10,000 rules evaluated every 30 seconds, the rule engine issues ~330 TSDB queries per second. Shard the rule engine: partition rules by rule_id (or by the service they cover) across multiple worker instances. Each worker evaluates its assigned rules independently. Alert state is stored in a shared store (Redis or Postgres) so workers can be restarted without losing state.

Deduplication. Two separate mechanisms work together here — they are easy to conflate but solve different problems:

1. State machine deduplication (local): the alert state machine fires a notification only on state transitions (OK → FIRING, FIRING → RESOLVED), not on every evaluation cycle. Once a rule is in FIRING state, the engine does not re-fire until it resolves and fires again. This prevents the "60 pages for a 30-minute outage" problem. The dedup key is (rule_id, label_set) — the same rule fires separately for different label values (e.g. service=payments vs service=auth), and each pages once.

2. Downstream delivery deduplication (distributed): the Notification Dispatcher delivers with at-least-once semantics — retries are possible. The downstream system (e.g. PagerDuty) deduplicates retried deliveries using a stable dedup_key (e.g. rule_id + label_set), not the local state machine. This ensures that a retry of an already-acknowledged notification is a no-op at the receiver side.

Grouping. A regional network event might trigger 500 rules simultaneously across all services in that region — if each pages individually, the on-call engineer receives 500 pages within 30 seconds, which is useless noise. A grouping policy collects alerts that share a common set of labels (e.g. region=us-east) over a short aggregation window (typically 30–60 seconds) and delivers them as a single grouped page: "47 alerts firing in us-east — top affected: payments, auth, user-service." Alertmanager (the open-source reference) calls this grouping and inhibition — inhibition suppresses downstream alerts when a root-cause alert is already firing (e.g. suppress all service alerts for a region once a "network-down: us-east" alert fires, so on-call sees one root cause instead of dozens of symptoms).

Reliable notification delivery. The Notification Dispatcher must deliver to external systems (PagerDuty, email, Slack) that are themselves sometimes down. Requirements:

• At-least-once delivery: persist every notification to a durable queue before attempting delivery. On failure, retry with exponential backoff.
• Delivery receipts: record when a notification was successfully acknowledged by the downstream system (PagerDuty webhook response, SMTP accepted). If the downstream never acknowledges, keep retrying for a bounded window, then escalate.
• Channel fallback: if PagerDuty is unreachable for more than N minutes, fall back to email or SMS for critical alerts. Configure the fallback in the notification channel definition.

Self-monitoring — the hardest part. An alerting system that fails silently is worse than no alerting system — you think you are covered, but you are not. Common failure modes:

• The rule evaluation engine crashes: no rules evaluate, no alerts fire, on-call has no idea.
• The TSDB is overloaded and rule queries time out: rules are not evaluated, alerts are missed.
• The notification queue is stalled: alerts are generated but pages are never delivered.

The canonical solution is a "dead man's switch" / watchdog alert: a synthetic metric (monitoring_heartbeat) is emitted every minute. An alert rule fires if this metric is absent for 2 minutes. If the alerting system is working, the rule evaluates and the synthetic metric is always present — no page. If the alerting system crashes, the metric disappears and the rule fires — but wait, if the system is down, who evaluates the rule? The watchdog alert must be evaluated by a separate, independent system (a simple external check, a second monitoring instance, or a hosted watchdog service like Deadmanssnitch or PagerDuty's built-in watchdog). This is the part most candidates miss.

Tradeoffs & bottlenecks

• Push vs pull for metric collection. Pull (Prometheus) is simpler to reason about (absent = visibly down) but struggles with ephemeral targets. Push works for any workload but requires heartbeat metrics to detect silent failures. Most production systems support both.
• Full-resolution raw storage vs rollups. Keeping raw data forever makes any query possible but storage compounds without bound. Rollups with retention tiers trade recall for cost and query speed. The right answer is both: raw short-term, coarser rollups long-term.
• Cardinality vs flexibility. Blocking high-cardinality tags is a real constraint — engineers cannot add user_id as a tag and must use distributed traces for per-user data. The tradeoff is system stability: an unconstrained cardinality explosion crashes the TSDB for all tenants.
• Alert engine fan-out vs isolation. Sharding the rule engine by service keeps one noisy service's rules from starving others' evaluation cycles, but adds coordination overhead for rules that cross service boundaries.
• At-least-once notification delivery vs dedup. At-least-once ensures pages are never silently dropped; combined with the state machine's dedup, a retry of an already-delivered page is a no-op (the rule is still in FIRING state and the dedup key is the same). The two mechanisms work together.
• Query cache TTL vs freshness. A 60-second query cache means a dashboard chart can be 60 seconds behind the latest data. For most operational dashboards this is fine; for a real-time SLO burn-rate alert evaluated every 30 seconds, cache the alert engine's queries separately at a lower TTL.

Extensions if asked

If the interviewer wants to push beyond the core system, add only the extension that changes the design discussion:

• Advanced percentile metrics. avg and sum roll up exactly; percentiles (p99) do not — you cannot compute p99 of p99s. The solution, used by the core design for latency series, is to store a histogram/sketch (a distribution of values in pre-defined or approximate buckets) and compute the percentile by merging histograms. Prometheus histograms and DDSketch address this.
• Distributed tracing and log correlation. Add a trace_id field to the alert state so engineers can jump from a firing alert directly to the traces and logs from the incident window. This requires a side-channel join between the TSDB and the trace store — traces are not metrics and should not be stored in the TSDB.
• Anomaly detection. Instead of static thresholds, evaluate rules against a dynamic baseline (rolling average ± N standard deviations). This requires maintaining a statistical model per series alongside the raw data. Start with a simple rolling-window baseline before reaching for ML.
• High availability of the monitoring system. Run two independent monitoring clusters (primary + standby) that both ingest and evaluate. Use a global router to send ingest to both and a consensus mechanism to avoid double-paging from two independent alert evaluations. This is the "quis custodiet" problem — your monitoring system itself needs to be monitored.
• Long-term cold storage. Rollups beyond 1 year (e.g. monthly averages for capacity planning) can be archived to columnar object storage (Parquet on S3) and queried by Athena or BigQuery. Uncommon for operational alerting but useful for capacity and cost planning.

What interviewers look for & common mistakes

What interviewers usually reward:

• Naming rollups and retention tiers early, not as an afterthought — and knowing to store min/max/sum/count (not just avg) in every rollup bucket so re-aggregation is accurate.
• Calling out the cardinality problem and proposing a concrete enforcement mechanism (per-metric series cap at ingest, blocking high-cardinality tag names) rather than ignoring it or saying "just use a good schema."
• Distinguishing the three paths (ingest, query/dashboard, alerting) and explaining each separately — many candidates blend them and miss the alerting-specific failure modes.
• Getting the alert state machine right — PENDING → FIRING with a duration guard, deduplication on state transitions (not on every evaluation cycle), and grouping for multi-alert outages.
• Mentioning reliable notification delivery (at-least-once, retries, channel fallback) and at-least-once is safe because the state machine handles dedup.
• The watchdog / self-monitoring problem — the alerting system must itself be monitored from the outside. This is the answer most candidates do not give.
• Rollup tier selection in the query path — explaining why a 30-day chart is fast (720 hourly buckets, not 172,800 raw points) is a clear signal that you understand what makes TSDBs work.

Before you finish, do a quick mistake check:

• Did you plan rollups and retention tiers from the start, not as a future optimization?
• Did you store min/max/sum/count in rollup buckets, not just the average?
• Did you call out cardinality explosion and propose a concrete ingest-time enforcement mechanism?
• Did you design the alert state machine with a PENDING → FIRING duration guard and deduplication on transitions?
• Did you address grouping (don't page 500 times for one regional event) and reliable notification delivery (at-least-once, retries)?
• Did you describe how the alerting system monitors itself — the watchdog problem?
• Did you explain why dashboard reads are fast (rollup tier selection, result cache, columnar TSDB layout)?
• Did you treat rollup data as derived and rebuildable from the raw data — never the source of truth?