Ranking survives the shift from algorithmic to random exposure. Calibration does not — and AUC never shows you that.
Try the exposure-bias calculator ↓ New here? Start with the plain-language tour →
The RankShift Serving series carries a stated limitation: offline lift was never validated against a randomized holdout. KuaiRand supplies one. Kuaishou inserted uniformly random videos into live recommendation feeds for two weeks, giving genuine missing-at-random data — the thing that makes unbiased offline evaluation possible.
The synthetic dataset the series was built on had a measured ceiling of 0.58–0.60 AUC, with only 2 of 25 candidate features separable from noise. On real logs, watch_ratio alone reaches 0.7486, and 24 of 37 request-time features clear the noise floor. The ceiling was a property of the broken joins, not of engagement prediction.
A feature counts as carrying signal when |AUC − 0.5| clears both three bootstrap sigma and an absolute floor of 0.005. The bootstrap resamples at 25%, so its standard error is rescaled to the full sample by √frac — without that correction the gate sits closer to six sigma than three, and roughly a third of the real features are discarded as noise.
The critical choice: both test sets cover the identical calendar window, 22 April – 8 May. Only the exposure mechanism differs. Any gap between them is exposure bias, not temporal drift.
Target is is_like ∨ is_comment ∨ is_forward ∨ is_follow. LightGBM, 400 rounds, 37 request-time features. Post-hoc signals (is_click, long_view, the *_stay_time pair) and all 50 daily video aggregates are excluded as label-contaminated.
A finding hiding in the base rates
Engagement under algorithmic exposure runs 3.80× higher than under random exposure — 2.121% against 0.557%. That ratio is a direct measurement of what Kuaishou's production recommender is worth against a uniform baseline, and it falls out of the data before any model is trained.
| Metric | A · algorithmic | B · random | Gap | Reading |
|---|---|---|---|---|
| ROC-AUC | 0.8811 | 0.8639 | +0.0173 | small, real |
| PR-AUC (raw) | 0.2865 | 0.1460 | +0.1405 | misleading — see below |
| Lift @ top 23.9% | 3.428× | 3.363× | +2.0% | transfers |
| Brier | 0.0175 | 0.0055 | — | base-rate bound |
| ECE | 0.0028 | 0.0060 | 2.1× worse | breaks |
| Mean pred ÷ actual | 0.96× | 2.07× | — | breaks |
The AUC gap of +0.0173 has a 200× paired-bootstrap 95% CI of [+0.0110, +0.0240] — separable from zero, but small enough that a team reporting only AUC would reasonably shrug at it.
Across the whole operating range, the two lift curves track each other. At the top 1% the model is better on randomly exposed traffic than on logged traffic.
This is the reassuring half. The decision the model is actually used for — which slice of traffic to spend the push budget on — is robust to exposure shift. The operating point chosen on logged data holds up on randomized data.
The same model, on the same window, with the same ranking quality, is systematically overconfident by 2.07× the moment exposure changes.
Which impressions to spend on. Lift at the operating point moves 2.0%; the top-1% slice actually improves.
How much engagement to forecast. Budget built on these probabilities would overspend by roughly half.
This matters precisely because the series stakes its argument on calibrated probabilities — on producing numbers that can enter a budget table and be summed, not merely ranked. That claim is correct on the training exposure distribution and wrong off it. The finding doesn't refute the thesis; it bounds it, which is a stronger place to argue from.
Raw PR-AUC drops from 0.2865 to 0.1460 — a 49% collapse. The obvious reading is that logged evaluation massively overstates performance.
That reading is backwards. PR-AUC is bounded below by the base rate, and the base rates differ by 3.80×. Normalizing each against its own split's difficulty inverts the conclusion:
| Split | PR-AUC | Base rate | PR ÷ base |
|---|---|---|---|
| A · algorithmic | 0.2865 | 0.0212 | 13.51× |
| B · random | 0.1460 | 0.0056 | 26.20× |
Against its own baseline the model is 1.94× better on randomly exposed traffic. The raw gap was almost entirely the base-rate difference wearing a metric's clothing.
Why this belongs in the writeup
It is the same failure mode as ranking four models by point-estimate AUC when their standard errors exceed the spread: a number that looks like evidence, compared across conditions where it isn't comparable. Catching it twice, in two unrelated places, is what makes it a habit rather than a lucky observation.
Fit a calibrator on nine days of randomized logs (22–30 April), apply it forward to the held-out eight (1–8 May). Fit and evaluation never share a day — this is the deployable scenario, not a shuffled split.
The fitted scalar comes out at 0.5012. The correction is: halve the probability.
| Calibrator | pred ÷ actual | ECE | Lift @ 23.9% | Brier gate |
|---|---|---|---|---|
| none | 2.111× | 0.00625 | 3.364× | — |
| scalar | 1.058× | 0.00051 | 3.364× | SHIP |
| platt | 1.054× | 0.00031 | 3.364× | SHIP |
| isotonic | 1.054× | 0.00031 | 3.364× | SHIP |
ECE improves 20×, and lift is identical to three decimals — a monotone transform cannot reorder, so calibration is free in ranking terms.
One caveat worth keeping: isotonic is only weakly monotone. It maps distinct scores onto shared values, and those ties move the metrics slightly — AUC 0.8639 → 0.8631 here, and lift 3.428× → 3.396× on served traffic. Scalar and Platt are strictly monotone and leave both untouched.
| Calibrator | pred ÷ actual | ECE | Lift @ 23.9% | Brier gate |
|---|---|---|---|---|
| none | 0.961× | 0.00284 | 3.428× | — |
| scalar | 0.481× | 0.01100 | 3.428× | reject |
| platt | 0.489× | 0.01084 | 3.428× | reject |
| isotonic | 0.491× | 0.01079 | 3.396× | reject |
The correction that fixes randomized traffic breaks served traffic by almost exactly the same factor in the other direction — 0.96× becomes 0.48×, and ECE degrades roughly fourfold.
ECE falls 20×, ranking untouched, Brier improves.
The identical transform, applied to the traffic actually served, overcorrects into a symmetric error.
The conclusion
Calibration is a property of the exposure policy, not of the model. One model cannot carry one calibrator across both distributions. Which calibrator is correct depends entirely on which traffic you intend to forecast — and that is a product question, not a modelling one.
The detail worth noticing: the Brier gate catches this without being told to. That gate was adopted in the original project for an unrelated reason — to stop ECE being gamed by collapsing predictions toward the mean. Here it independently reads SHIP on one distribution and reject on the other, with no special-casing. A guardrail built for one failure mode caught a second one it was never designed for.
The random log makes the quantity OPE methods estimate — a policy's true value — directly measurable. So instead of trusting an estimator, grade all of them: estimate the uniform policy's value from the standard log alone, using item-level propensities (show frequency, Laplace-smoothed), then check the answer.
| estimator | estimate | × truth | 95% CI covers truth? |
|---|---|---|---|
| naive average | 0.02121 | 3.80× | no |
| IPS | 0.01671 | 3.00× | no |
| SNIPS | 0.02058 | 3.69× | no |
| direct method | 0.02145 | 3.85× | no |
| doubly robust | 0.02124 | 3.81× | no |
Every estimator misses, including doubly robust — and not from variance. The effective sample size after weighting is 42,633 and every interval is tight. The propensity model is wrong in a way reweighting cannot repair, and DR inherits the direct method's bias because the reward model was itself trained on biased exposure.
The reverse direction diagnoses it. Estimating the standard policy's value from the random log — where logging propensities are exactly uniform — an item-marginal target recovers 0.00694 of a true 0.02121: 33%. With propensities exact, the missing two-thirds is purely the item-marginal approximation, which decomposes the exposure bias:
The decomposition
3.80× = 1.25× × 3.04×. The standard policy's measured advantage over uniform splits into 1.25× from which videos it picks — the item marginal, the part popularity-corrected estimators can see — and 3.04× from which user it shows them to. Matching, not selection, is where the value lives, and item-level propensities cannot express it. If the logger does not record propensities, no estimator reconstructs them afterwards.
One contract detail: the reward model here uses only user and video features. A counterfactual (u, v) pair has no watch time — it never happened — so an OPE reward model lives under a stricter feature contract than the serving model, and reusing the serving model would smuggle an outcome into a counterfactual.
Set how much better a recommender is than uniform exposure — split into which videos it picks and which user it shows them to — then choose what your logs recorded about propensities. The bars show what each estimator would report for the uniform policy's true value. Defaults are the KuaiRand measurements from Finding 05.
Measured on KuaiRand-Pure, item-level propensities: naive 3.80× · IPS 3.00× · SNIPS 3.69× · direct method 3.85× · doubly robust 3.81×. The calculator is a stylized model of that mechanism — item-level weights remove the selection factor and none of the matching factor — not a re-run of the experiment.
Finding 04's numbers, as a choice. Pick where the scalar calibrator was fitted and where it is deployed.
"none" and "fitted on randomized" are measured (experiment 03, scalar factor 0.5012). "Fitted on served" is inferred as the scalar 1/0.961 applied to both measured baselines.
Two PyTorch models on the same splits and metrics. A two-tower with the request-time features through an MLP head — feature parity with the LightGBM — and the pure retrieval two-tower: id embeddings and profiles only, dot product, no watch signal by construction.
| model | TEST-A AUC | TEST-B AUC | TEST-A lift |
|---|---|---|---|
| LightGBM | 0.8811 | 0.8639 | 3.428× |
| two-tower + request features | 0.8815 | 0.8519 | 3.437× |
| two-tower (pure retrieval) | 0.8246 | 0.8007 | 3.035× |
At feature parity the neural ranker ties the GBDT — on a million rows of tabular features that is the literature's expected result, reported rather than tuned away. The more informative number is the pure tower: 0.8246 with no watch signal at all. Personalization alone carries most of the ranking power — the AUC-level restatement of Finding 05's decomposition.
KuaiRand-27K: the same users against the full 32M-video catalogue — 322M interactions, ~48 GB unpacked, more than this machine's disk or RAM. The tarball is streamed once into column-pruned parquet (9.9 GB → 3.3 GB, raw CSV never lands), DuckDB joins out of core, LightGBM fits on an 8M-row reservoir sample, and the full 186M-row test window is scored exactly, in chunks.
| Pure · 2.6M | 27K · 322M | ||
|---|---|---|---|
| lift, A → B | 3.428× → 3.363× | 3.702× → 3.483× | ranking transfers |
| pred ÷ actual, A → B | 0.96× → 2.07× | 0.99× → 2.29× | calibration breaks |
| TEST-A AUC | 0.8811 | 0.9147 | more data helps |
Both headline findings replicate at two orders of magnitude more data — and the calibration break is slightly larger at scale. More data sharpens the model's fit to the exposure policy it was trained under, which is precisely the thing that does not transfer.
play_time_ms and watch_ratio are treated as request-time features, matching the original service's contract, where watch duration arrives with the request. In a pre-impression ranking setting they would be unavailable and every number above would fall.The open thread is the leakage fingerprint. Among the 50 daily video aggregates, exposure-side counts carry no signal — show_cnt 0.5031, play_cnt 0.5049 — while engagement-side counts carry a great deal: like_cnt 0.6282, double_click_cnt 0.6328. Genuine popularity features would move together. That they don't is a signature of label contamination, and it generalizes into a reusable test: for any aggregate feature family, exposure-side and engagement-side columns must carry comparable signal; if only the engagement side does, the family is contaminated.
The calibration result also has a live question underneath it. Serving traffic is algorithmically exposed, so the uncalibrated model is already correct for today's forecast. The randomized calibrator earns its place the moment the exposure policy changes — a new ranker, an exploration bucket, a cold-start slice. Deciding which of the two is in force is a product question, and it should be an explicit one.
The model these experiments interrogate is served by four small services — training and single prediction, batch scoring with per-row fault isolation, artifact introspection, and request analytics with drift alarms — collectively RankShift Serving. All four are on this site as a plain-language tour: start here →. The single and batch pages run the shipped model's parameters in your browser; model info and analytics are captures from the running APIs.