Design an Ads Ranking System

ML Ranking & Calibration System Design

Overview

Every time you open Snap, Instagram, or a search results page, a hidden auction runs in the time it takes the screen to paint. The platform pulls a set of eligible ads, predicts how likely you are to click and convert on each one, multiplies those predictions by what each advertiser is willing to pay, and picks the ads that maximize a blend of revenue and user experience — all in a few tens of milliseconds, billions of times a day.

This is a canonical MLE / ML-system-design question, and it is deceptively different from organic feed ranking. In an organic recommender, the model's job is to order items well: if the true best clip ends up on top, it does not matter whether the score was 0.9 or 0.4. In ads, the score gets multiplied by money. The ranking value of an ad is roughly eCPM ≈ bid × pCTR (times pCVR for conversion goals), so a miscalibrated pCTR does not just reorder ads, it distorts the auction, mischarges advertisers, and burns budgets at the wrong rate. AUC can't catch this: it depends only on rank order, so multiplying every pCTR by 2 (which doubles every eCPM) leaves AUC unchanged — which is why log-loss / ECE, not AUC, gate an ads model. That is why calibration, not just ranking quality, is the beating heart of this system.

The two interview audiences push in different directions:

• SWE interviewer: drills the systems spine — the request path from ad request to rendered ad, how feature hydration works under a tight budget, how the auction and pacing services fit, how logs flow back to training.
• MLE interviewer: drills the modeling — the multi-task architecture, why predictions must be calibrated probabilities, how you train on delayed conversion labels without teaching the model that every fresh impression is a negative, and how you keep calibration from drifting.

The flow to keep in your head: request → candidate ad retrieval → feature hydration → ranker scoring → filtering/policy → auction & pacing → serve → logging. That last arrow — logging — closes the loop and produces tomorrow's training data.

Out of scope (say so): the advertiser-facing campaign creation UI, creative asset storage and moderation, billing and invoicing internals, and brand-safety classification. We own the online ranking + auction path and the training loop that feeds it.

Functional requirements

• Rank candidate ads for a request. Given an ad request (user + context + a slot to fill), select and order a set of eligible ads and return the winners to render.
• Predict multiple engagement signals. For each candidate ad, predict click, conversion, and negative signals (hide/report) so ranking reflects more than a raw click.
• Run the auction. Combine each ad's predictions with its advertiser bid into a comparable ranking value (eCPM), pick winners, and compute what each winner pays.
• Respect budgets via pacing. Spend each advertiser's daily budget smoothly across the day rather than exhausting it in the first hour.
• Enforce policy and filtering. Drop ads that are ineligible for this user (frequency caps, targeting, blocked categories, policy violations) before or after scoring.
• Log everything for training. Record each impression, its features at serving time, the winning price, and later the click and conversion events, so the training loop can reconstruct exactly what was shown and why.

Out of scope (state it): campaign setup, creative moderation, advertiser dashboards, fraud/invalid-traffic detection (mention it as a filter), and billing reconciliation.

Non-functional requirements

• Tight latency. The ad response shares the page's budget with organic content — target <150ms end-to-end at p99 for the ads path. Budget ~30ms for retrieval, ~40ms for feature hydration, ~50ms for ranking inference, ~20ms for auction + pacing + response.
• High throughput. Hundreds of thousands of ad requests per second at peak, each scoring hundreds of candidates.
• Calibration is a hard requirement, not a nice-to-have. Because predictions are multiplied by money, the model's p=0.02 must actually mean a 2% event rate. Miscalibration directly mischarges advertisers and misallocates budget.
• Availability over precision. If the ranker is slow or a feature is missing, fall back to a simpler model or default feature values. Serving a slightly worse ad beats serving a blank slot (lost revenue) or an error.
• Revenue and user experience are dual objectives. Maximizing short-term revenue by flooding the best slots with ads degrades retention. The system optimizes long-term value, which constrains ad load and penalizes negative feedback.
• Auditability. Log enough (user, candidates, features, scores, price, pacing state) to reconstruct any charge and any ranking decision — required for advertiser trust, billing disputes, and debugging regressions.

Estimations

State assumptions; the goal is to size the subsystems, not to be exact.

Rounded assumptions

• Users: ~300M daily active users; each sees many ad slots per day across sessions.
• Ad requests/sec: assume ~30B ad opportunities/day ÷ 86,400s ≈ ~350K ad requests/sec; peak is 2–3×, so ~800K QPS at peak.
• Eligible ads (corpus): millions of active ads at any time; targeting + policy prune this to ~500–1,000 candidates per request after retrieval.
• Latency budget: total p99 ~150ms — retrieval ~30ms, feature hydration ~40ms, ranking inference ~50ms, auction + pacing + assembly ~20ms.
• Training data volume: ~30B impressions/day, each logged with a feature snapshot; clicks arrive within seconds–minutes, conversions arrive over hours to days (attribution windows of 1–7+ days).
• Label sparsity: click rates are low (~1–3%); conversion rates are far lower (fractions of a percent), so the positive class is rare and delayed.

How the numbers affect the design

Core entities / data model

Seven things are modeled: the users and ads (first-class objects), the campaign/bid economics, the feature entries that power scoring, the auction outcome, and the two log streams that close the training loop.

The impression log is the training-data factory. Every served ad is logged with the feature values used at serving time — not recomputed later. This is what prevents training/serving skew, and it also snapshots the pacing and position context needed to debug the auction later.

Clicks and conversions are separate, differently-timed facts. A click is joined to its impression within minutes. A conversion is joined via an attribution window that can span days — and the same impression can accumulate a conversion label long after training normally would have run.

Where each lives. User/ad/campaign metadata in a sharded store (partitioned by id). The online feature store in an in-memory KV store (Redis/Memcached) for sub-10ms reads. Campaign budget/pacing state in a fast, strongly-consistent counter store (spend must not race). Impression and conversion logs in a high-throughput event stream (Kafka) feeding a warehouse (BigQuery/Hive).

API design

Four operations carry the system: request ads, log the impression, log the click, and log the conversion. The internal retrieval/rank calls sit behind the ad-request handler.

Request ads for a slot

POST /api/ads/request
Body: {
  "user_id": "...",
  "context": { "surface": "feed", "slot": 2, "device": "ios", "hour": 14 },
  "num_ads": 1
}
200 OK
Returns: {
  "ads": [ { "ad_id": "...", "creative": {...}, "price_cpm": 4.20, "click_id": "clk_..." } ],
  "request_id": "req_abc123"   ← correlates impression + click + conversion logs
}

The request_id (and a signed click_id per ad) is returned so downstream click and conversion events can be joined back to exactly what was shown.

Log an impression (batched, async)

POST /api/ads/impressions
Body: { "request_id": "req_abc123",
        "impressions": [ { "ad_id": "...", "rank_position": 0, "shown_at_ms": 172... } ] }
202 Accepted

Log a click

POST /api/ads/click
Body: { "click_id": "clk_...", "request_id": "req_abc123", "ad_id": "...", "ts_ms": 172... }
202 Accepted

Log a conversion (arrives hours–days later, from advertiser pixel/SDK)

POST /api/ads/conversion
Body: { "click_id": "clk_...", "ad_id": "...", "conversion_type": "purchase",
        "value": 39.99, "ts_ms": 172... }
202 Accepted

Internal: retrieve candidates

POST /internal/retrieve
Body: { "user_id": "...", "context": {...}, "limit": 1000 }
Returns: { "candidates": [ { "ad_id": "...", "campaign_id": "..." }, ... ] }

Internal: score candidates

POST /internal/rank
Body: { "user_id": "...", "candidates": ["ad_1", ...], "context": {...} }
Returns: { "scores": { "ad_1": { "pCTR": 0.021, "pCVR": 0.004, "p_hide": 0.001 }, ... } }

Note the ranker returns raw calibrated probabilities per task, not a single blended number — the auction service combines them with the bid, because only the auction knows each campaign's bid and objective.

High-level architecture

Two paths run in parallel: the serving path (ad request → auction → served ad in <150ms) and the training/data path (impressions + delayed labels → retrained, recalibrated models). A strong answer walks both.

1. Serving path — request → retrieval → ranking → auction → serve

flowchart LR
    Client["Client"] -->|"POST /ads/request"| AdSvc["Ad Request Service"]
    AdSvc -->|"user + context features"| FS[("Online Feature Store")]
    AdSvc -->|"eligible ads"| Retr["Retrieval / Targeting"]
    Retr -->|"~1000 candidates"| AdSvc
    AdSvc -->|"hydrate ad + cross features"| FS
    AdSvc -->|"score candidates"| Rank["Ranking Service (multi-task model)"]
    Rank -->|"pCTR, pCVR, p_hide"| AdSvc
    AdSvc -->|"eCPM per ad"| Auction["Auction + Pacing"]
    Auction -->|"winner + price"| AdSvc
    AdSvc -->|"served ad"| Client
    AdSvc -.->|"log impression async"| Q[["Kafka: Impressions"]]

1. The Ad Request Service reads user + context features from the online feature store in one batched call.
2. Retrieval/targeting prunes millions of active ads to ~1,000 eligible candidates (targeting match, frequency caps, policy).
3. Ad + cross features for those candidates are hydrated from the feature store under the latency budget.
4. The Ranking Service scores all ~1,000 candidates in one batched forward pass on GPU (with a lightweight logistic-regression fallback if the deep model is slow or unavailable), returning calibrated per-task probabilities (pCTR, pCVR, p_hide).
5. The Auction + Pacing service converts each to eCPM using the campaign bid, applies pacing multipliers, picks winners, and computes prices.
6. Impression logging is fully async and never blocks the response.

Retrieval and feature hydration under budget. Two systems dominate the pre-ranking latency. Retrieval runs two arms in parallel: (1) a targeting/eligibility index keyed by geo, interest, audience lists, and frequency caps for deterministic match, intersected with the request and pruned by policy filters; and (2) an embedding-based recall arm (two-tower-style) for lookalike / broad-match candidates the deterministic index would miss. Feature hydration then batch-fetches the ad, advertiser, and (user, ad) cross features for all candidates in a single multi-get against the online store — never N sequential reads. Missing features fall back to defaults rather than blocking, and the hydrated snapshot is exactly what gets logged, so training sees what serving saw. Real-time features flow in on a separate path: session-level signals (last-N clicks/queries this session) stream via Kafka/Flink into the online feature store so intent — e.g. the user just searched "running shoes" — is reflected immediately rather than at the next retrain.

2. Training / data path — impressions + delayed labels → model + calibration

flowchart LR
    Serve["Ad Serving"] -->|"impression + feature snapshot"| ImpQ[["Kafka: Impressions"]]
    Client2["Client / Advertiser pixel"] -->|"clicks + conversions"| LblQ[["Kafka: Labels"]]
    ImpQ --> Joiner["Label Joiner (wait out attribution window)"]
    LblQ --> Joiner
    Joiner -->|"impression + matured labels"| DW[("Data Warehouse")]
    DW -->|"training set"| Train["Training Pipeline (multi-task ranker)"]
    Train -->|"model artifact"| Reg[("Model Registry")]
    DW -->|"held-out logs"| Calib["Calibration Fitter (Platt / isotonic)"]
    Calib -->|"calibration maps"| Reg
    Reg -->|"hot-reload weights + calib"| Rank["Ranking Service"]

The label joiner is the crux: it holds impressions in a buffer, joins clicks quickly and conversions across a long attribution window, and emits training rows only once labels have matured — while correcting for the fact that a not-yet-converted impression is not a confirmed negative (Deep Dive 2). A separate calibration fitter re-fits calibration maps on recent held-out logs and ships them alongside the model (Deep Dive 3).

3. Combined — the full picture

flowchart LR
    User["User"] --> App["Client"]
    App -->|"ad request"| AdSvc["Ad Request Service"]
    AdSvc --> FS[("Feature Store")]
    AdSvc --> Retr["Retrieval"]
    AdSvc --> Rank["Ranking Service"]
    AdSvc --> Auction["Auction + Pacing"]
    Auction --> App
    App -.->|"impressions + clicks + conversions"| Kafka[["Kafka"]]
    Kafka --> Join["Label Joiner"]
    Join --> DW[("Data Warehouse")]
    DW --> Train["Training + Calibration"]
    Train --> Reg[("Model Registry")]
    Reg --> Rank
    DW --> FS

The serving path is synchronous and latency-critical; the training path is asynchronous and throughput-critical. The feature store bridges them: offline jobs precompute features that serving reads at low latency, using shared computation to avoid skew.

Deep dives

1. The multi-task ranking model and how scores enter the auction

The ranker takes a (user, ad, context) feature vector and predicts several outcomes at once. Ads have different objectives — some campaigns pay for clicks, others only value conversions (purchases, installs, signups) — and the same impression carries positive and negative signals. A single CTR model cannot serve all of this.

Feature groups. Four families feed the model: user (demographics, behavioral embeddings, interests), ad (creative type, historical CTR, advertiser category, bid, objective), context (surface, position, device, hour), and cross (user×ad affinity, user×category engagement).

Multi-task architecture. The strong answer is a multi-task model with a shared bottom and per-task heads:

flowchart LR
    Feats["User + Ad + Context features"] --> Shared["Shared bottom (embeddings + MLP)"]
    Shared --> H1["Head: pCTR"]
    Shared --> H2["Head: pCVR"]
    Shared --> H3["Head: p_hide"]
    Shared --> H4["Head: downstream value"]

• Shared-bottom is the simplest: one shared MLP, several linear heads. Cheap, but tasks can interfere.
• MMoE (Multi-gate Mixture-of-Experts) replaces the single shared bottom with several expert sub-networks and a per-task gate that mixes them, letting each task draw on different experts and reducing negative transfer between weakly-correlated tasks (clicks vs long-delay conversions). PLE (Progressive Layered Extraction) is the successor, adding task-specific extraction layers alongside the shared experts to separate shared from task-private representations more cleanly.

A weighted logistic regression is a perfectly respectable baseline to name first — massive sparse feature crosses, one model per task, trained on billions of rows. Deep multi-task models add learned embeddings and feature crosses on top; mention the baseline, then justify the upgrade.

Per-task label sparsity is asymmetric. The click head trains on ~1–3% positives; the conversion head on fractions of a percent, and those positives arrive late (Deep Dive 2). This asymmetry is why multi-task helps — the dense click signal regularizes the shared representation that the sparse conversion head reuses. It is also why you typically downsample negatives to make training tractable (keep all positives, sample negatives), which shifts the base rate and therefore mandates a calibration correction afterward (Deep Dive 3). A "weight" per task in the loss compensates for the sampling and for the relative business value of each signal.

Cold-start for new ads. A brand-new ad or advertiser has no historical pCTR, so its estimate is prior-based (content + advertiser priors) and highly uncertain. Explore that uncertainty deliberately — Thompson sampling or UCB on the pCTR posterior — so new ads get shown enough to gather signal instead of being starved by better-known competitors.

How scores enter the auction. The heads' outputs are combined with money, not just weights:

eCPM(click goal)      = bid_per_click × pCTR × 1000
eCPM(conversion goal) = bid_per_conversion × pCTR × pCVR × 1000

For a conversion campaign, the ad only earns money if the user clicks and converts, so its expected value chains pCTR × pCVR. The auction ranks all heterogeneous ads by this common eCPM currency, then charges the winner its critical price — the minimum bid at which it still would have won:

price_per_click = (runner_up_eCPM / winner_pCTR) / 1000

This is the industry-standard generalized second-price (GSP) mechanism. Note the subtlety: pure second-price (Vickrey / VCG) is incentive-compatible — truthful bidding is optimal — but GSP is not truthful; it is the standard multi-slot mechanism precisely because VCG is hard to run at scale, not because it shares VCG's truthfulness. Calibrated predictions still matter because they keep the critical-price computation and pricing accurate, regardless of the mechanism's incentive properties.

The ranking value also carries user-experience terms, not just advertiser value. Subtracting a probability from an eCPM (dollars) is a dimensional error, so use value = eCPM − λ·p_hide where λ is a tuned penalty in eCPM units (or a multiplicative discount eCPM·(1 − γ·p_hide)). A per-request reserve price then floors the auction so a weak ad loses to showing organic content. This is how the system trades short-term revenue against long-term retention inside a single comparable score.

2. Delayed and sparse conversion labels

Clicks are easy: they arrive within seconds and join cleanly to the impression. Conversions are the hard label. A user clicks an install ad today and installs the app three days later; a purchase can land anywhere inside a 1-, 7-, or 28-day attribution window. At training time, most recent impressions have no conversion yet — and the fatal mistake is to call that a negative.

The maturity problem. You face a bias either way:

• Wait for the full window (e.g. 7 days) before training on any impression → labels are correct, but the model always trains on 7-day-old data and cannot react to fresh trends. Freshness suffers.
• Train immediately on fresh impressions → freshness is great, but positives that haven't arrived yet are counted as negatives. Labels are wrong.

Delayed-feedback / delayed-label correction resolves the tension so you can train on fresh data and stay unbiased:

flowchart LR
    Imp["Fresh impression"] --> Train["Train now (freshness)"]
    Train --> Corr["Delayed-feedback correction"]
    Late["Conversion arrives later"] --> Update["Positive relabel / weight update"]
    Corr --> Model["Unbiased pCVR"]
    Update --> Model

• Delayed-feedback model (Chapelle). Jointly fit conversion probability and an exponential delay distribution; the elapsed-time term lets a not-yet-converted impression contribute correctly rather than being hard-labeled 0. The importance-weighted "fake-negative" variant below is a separate, later formulation — not the same thing.
• Positive-unlabeled (PU) learning. Frame the data as positives + unlabeled (not positives + negatives). Recent unconverted impressions are unlabeled, and PU estimators recover the true positive rate under a known/estimable label-frequency (class-prior) assumption, without treating them as clean negatives.
• Fake-negative / positive relabeling (importance-weighted). Ingest recent impressions as tentative negatives for freshness, then when a delayed conversion arrives, emit a correction event that flips the label and adjusts the loss (importance-weighted) so the earlier negative is undone.
• Negative-label maturity. Only let an impression become a confirmed negative once its attribution window has fully closed with no conversion. Before that, it is unlabeled, not negative.

Online proxy metrics. Because true pCVR quality is only knowable after windows close, watch proxy signals in near-real-time so a bad model doesn't run for a week undetected: predicted-vs-observed conversion ratio on already-matured cohorts, calibration on the last fully-closed day, click-to-early-conversion rate, and revenue/spend guardrails. If predicted conversions run 30% above matured actuals, pull the model before the full window even confirms it.

3. Calibration, auction compatibility, and pacing

Calibration is where ads ranking diverges hardest from organic ranking. In an organic feed, an uncalibrated-but-well-ordered model is fine. Here, three downstream systems read the probability literally:

• The auction: eCPM ∝ bid × pCTR (× pCVR) — a 20% calibration error is a 20% pricing error on every affected auction.
• Pacing: budget pacing predicts spend rate from expected win rate × price; wrong probabilities mean budgets exhaust too early or go unspent.
• Billing: cost-per-action bidding backs out a per-impression bid from pCVR, so miscalibration directly over- or under-charges advertisers.

Techniques. Calibration is a lightweight post-hoc map fit on held-out logs, applied to the raw model output:

• Platt scaling — fit a logistic function (sigmoid) mapping raw scores to probabilities. Cheap, few parameters, good when miscalibration is a smooth monotonic squash.
• Isotonic regression — fit a free-form monotonic step function. More flexible than Platt, handles non-sigmoid distortion, but needs more data and can overfit in sparse regions (a real risk for rare conversions).
• Calibration by segment. A single global map hides per-slice miscalibration. Fit separate maps per segment — surface, country, device, objective, new-vs-established advertiser — because a model well-calibrated in aggregate can be badly off for, say, a new advertiser with little data.

flowchart LR
    Raw["Raw model score"] --> Cal["Calibration map (per segment)"]
    Cal --> Prob["Calibrated pCTR / pCVR"]
    Prob --> Auction["Auction (eCPM ∝ bid × p)"]
    Prob --> Pace["Pacing"]
    Prob --> Bill["Billing"]

Calibration drift. Calibration is not fit-once. The traffic mix, the ad corpus, and user behavior all shift; a map fit last week can be stale today. Re-fit calibration on recent held-out logs on a fast cadence (hourly/daily), independent of the slower full-model retrain, and monitor observed-vs-predicted ratio as a live guardrail.

Budget pacing reads those calibrated spend predictions. Pacing is a functional requirement, so design it, don't hand-wave it. Each campaign's remaining_budget is decremented from confirmed spend in a strongly-consistent counter store. A pacing multiplier in [0, 1] then throttles the campaign's auction participation — either gating whether it enters an auction or scaling its bid down — so spend tracks a smooth target curve across the day instead of front-loading. Run it as a control loop (PID-style): compare actual spend against the ideal spend-so-far, and nudge the multiplier up when under-pacing and down when over-pacing.

flowchart LR
    Spend["Confirmed spend"] --> Budget[("remaining_budget counter")]
    Budget --> Ctrl["Pacing controller (PID vs target curve)"]
    Ctrl -->|"pacing multiplier"| Auction["Auction (scale bid / gate entry)"]
    Auction -->|"new spend"| Spend

Tradeoffs & bottlenecks

• Single-task CTR vs multi-task. A single CTR model is simpler and lower-latency but cannot serve conversion objectives or penalize negative feedback; multi-task (shared-bottom → MMoE/PLE) serves all objectives at the cost of training complexity and negative-transfer risk.
• Weighted logistic regression vs deep model. Logistic regression with hand-crafted crosses is cheap, interpretable, easy to calibrate, and a strong baseline at this scale; deep models capture interactions automatically and lift accuracy but cost more latency and are harder to keep calibrated.
• Retrieval vs full ranking. Retrieval trades a little recall (a good ad occasionally pruned before scoring) for the ability to hit the latency budget; widen the candidate set for more recall at the cost of more hydration + inference compute.
• Freshness vs label correctness. The central tension of Deep Dive 2 — train on fresh impressions (risking mislabeled delayed positives) vs wait for matured windows (staleness). Delayed-feedback correction buys both at the cost of pipeline complexity.
• Offline vs online metrics. The ads-standard offline set is Normalized Entropy (NE), AUC-PR (preferred over AUC-ROC when positives are rare), and Expected Calibration Error (ECE) — necessary but insufficient, since a model can win on AUC yet lose revenue or hurt user metrics. The truth is the online A/B test on revenue, advertiser ROI, and user-experience guardrails (ad load, hide rate, retention).
• Pacing lag. Pacing reacts to spend that is itself measured with delay; overly aggressive pacing under-spends budgets, overly loose pacing front-loads them. It is a control-loop tuning problem, not a one-time setting.
• Calibration is silent when it breaks. Like training/serving skew, miscalibration throws no error — it just quietly misprices auctions. Live observed-vs-predicted monitoring per segment is the only defense.

Extensions if asked

• Pacing as a dual variable. Beyond the PID control loop of Deep Dive 3, pacing can be framed as a dual-variable on the budget constraint, throttling participation to solve the constrained-optimization view of smooth spend.
• Feature and embedding stores. A dedicated feature store with shared online/offline computation; embedding stores for high-cardinality ids (user, ad, advertiser) that both the ranker and retrieval consume.
• Debugging a live regression. When revenue drops, systematically check: calibration drift (observed-vs-predicted), a feature pipeline break (skew detection), a bad model rollout (compare A/B arms), or a pacing misconfiguration — in that order, because each has a distinct signature in the logs.

What interviewers look for & common mistakes

What interviewers usually reward:

• Naming the retrieval → ranking → auction funnel immediately, and stating that scores feed an auction that multiplies them by money — so calibration, not just ordering, is required.
• The eCPM chain. eCPM ≈ bid × pCTR for clicks, × pCVR for conversions, and why both heads must be calibrated for the chain to price correctly.
• Multi-task modeling with per-objective heads, the weighted-logistic-regression baseline, and MMoE/PLE to reduce negative transfer between click and conversion tasks.
• Delayed conversion labels handled correctly — recognizing that "not converted yet" is not a negative, and proposing a concrete correction (delayed-feedback model, PU learning, positive relabeling) plus online proxy metrics.
• Calibration as a first-class stage — Platt/isotonic, per-segment maps, drift monitoring — and awareness that position and selection bias distort the labels feeding it.
• A concrete latency budget across retrieval, hydration, ranking, and auction, and a fallback story when a stage is slow or a feature is missing.

Before you finish, do a quick mistake check:

• Did you say calibration matters, not just ranking/AUC, and explain that the auction multiplies predictions by money?
• Did you avoid treating not-yet-converted impressions as negatives, and name a delayed-label correction?
• Did you address position and selection bias distorting the training/calibration labels?
• Did you use a multi-task model with per-objective heads rather than a single CTR model when goals differ?
• Did you snapshot features at serving time to prevent training/serving skew?
• Did you propose exploration so new ads get signal instead of never being shown?
• Did you name the online A/B test (revenue + user guardrails) as the true evaluation signal, not offline AUC alone?