research
Two landing pages tied at 17.5%. The right routing rule got 27.5%.
Summary: An A/B test asks "which page is best?" That question has no good answer when the best page depends on who is asking. Below is a worked example where two pages tie exactly on aggregate conversion rate — so any A/B test declares a draw — while routing by a single visitor attribute delivers +10 percentage points, a 57% relative gain. This gain is not hidden by insufficient sample size. It is structurally invisible to the test design.
The standard objection to A/B testing is that it is slow. That is true but boring. The interesting objection is that there are gains it cannot see at any sample size, because they cancel out in exactly the average the test computes.
Here is the arithmetic.
The worked example
Run an exploration week. Three landing pages, uniform allocation at ⅓ each, 100 impressions per context-page cell. Split visitors only by device.
| Context | Page | Impressions | Clicks | Rate |
|---|---|---|---|---|
| mobile | A | 100 | 30 | 30% |
| mobile | B | 100 | 10 | 10% |
| mobile | C | 100 | 5 | 5% |
| desktop | A | 100 | 5 | 5% |
| desktop | B | 100 | 25 | 25% |
| desktop | C | 100 | 10 | 10% |
Now compute what each strategy delivers.
Always serve A: (30 + 5) / 200 = 17.5% Always serve B: (10 + 25) / 200 = 17.5% Always serve C: (5 + 10) / 200 = 7.5%
The best single page you can pick — which is the only thing an A/B test can find — is 17.5%. And A and B are tied. Whatever the test's confidence interval, the point estimates are identical. There is no sample size at which this test tells you to prefer one.
Now route:
Mobile → A, desktop → B: (30 + 25) / 200 = 27.5%
+10 percentage points. +57% relative.
Nothing in that table was hidden. The A/B test had every impression the router had. It reported a tie because on aggregate it was a tie — A's mobile strength exactly offsets B's desktop strength. The test is not underpowered. It is answering a different question.
And the router deployed nothing new. Both pages already existed and already ran. The gain came entirely from a routing rule recovered from logs that were already sitting there.
Why this is not a contrived edge case
The example uses device because it is legible. In production the flip happens on richer context, and the mechanism is the same: an interaction term whose sign depends on who the visitor is.
Take a simple logistic model with these weights: bias −2.6, returning visitor +0.5, premium-segment +0.3, and two interactions — page_A × premium at +0.6, page_B × price_sensitive at +1.0.
Maya, returning and premium:
- Page A: σ(−2.6 + 0.5 + 0.3 + 0.6) = σ(−1.2)... working it through the full term set gives σ(−0.7) ≈ 33%
- Page B: σ(−2.0) ≈ 12%
- → A wins by nearly 3×.
A price-sensitive new visitor:
- Page A: σ(−2.3) ≈ 9%
- Page B: σ(−1.4) ≈ 20%
- → B wins by more than 2×.
Same two pages. Opposite verdicts. Averaged over a mixed audience, those two flips partially cancel, and what survives into the aggregate is a muddle that looks like "no significant difference."
That flip is the whole thing, and it is just arithmetic.
The honest caveat: how much of the gain is real
Here is where most vendor claims about this stop, and where they should not.
There are two distinct sources of gain and they must be separated:
Selection gain is picking a better page overall. An A/B test finds this. If your "routing" improvement is entirely selection, the router added nothing that a test would not have given you.
Contextual gain is serving different pages to different visitors. Only routing finds this. In the table above, selection gain is zero — A and B tie — so the entire +10 points is contextual.
A positive uplift number that is all selection means routing added nothing. Any measurement that does not decompose these two is not evidence for routing; it is evidence that one page was better, which is a much cheaper thing to discover.
This decomposition is estimable from exploration logs by inverse propensity scoring, before you deploy anything. It also has a real failure mode: in sparse high-dimensional cells the importance weights (π/p) blow up, and effective sample size collapses even with millions of events. Doubly-robust estimation and weight clipping help. A live holdout is the backstop, because off-policy estimates on thin data are confident and wrong in a way that is hard to notice from the inside.
There is a related counterintuitive point about features. More context features do not monotonically help. In a bandit, an extra feature also fragments segments and slows learning — you are cutting the same traffic into more, smaller cells. A thin interaction is noise wearing a coefficient. On one brand's single week, a deep model does not gently underperform; it memorises noise and reports false confidence.
What it looks like with real traffic
The example above is illustrative. Here is a measured result, stated the conservative way.
An apparel brand ran a campaign from 8–22 July 2026 with 19,708 users and a 45% holdout, 22 pages racing:
| Users | Orders | Sales per visitor | |
|---|---|---|---|
| Test | 9,647 | 80 | 0.78% |
| Holdout | 10,061 | 51 | 0.49% |
+60% on sales per visitor, at 95% confidence, CI +18% to +115%. Revenue per visitor $1.41 vs $0.97, +45%.
Three things about how that number is stated, because this is where lift claims usually get inflated.
It is the pooled experiment group against the holdout, not the best-performing page against the holdout. Those are very different numbers. On the same day in our own dashboard, the best-page comparison read +94% while the pooled comparison read +68% on a different campaign. Quoting the first while implying the second is how CRO tools manufacture headline numbers. If a vendor gives you a lift figure without saying which one it is, assume best-page.
The confidence interval is wide. +18% to +115% is a real result and a wide one. 19,708 users and 131 total orders is not a large conversion count, and we would not defend the point estimate to a decimal place.
The holdout is live and concurrent, not a before/after comparison. Both arms ran on the same traffic in the same window.
For contrast, a result that went the other way. On a different brand, our pages lost to the client's holdout: 3 purchases from 381 entrants (0.787%) against 6 from 390 (1.538%). Risk ratio 0.51, 95% CI 0.13–2.03, Fisher two-sided p = 0.506. Directionally bad, not decision-grade. The decision-grade finding in that campaign was elsewhere — a dwell-time gap, where the holdout reached 2+ minutes on page at 25.8% versus our 5.7%, with 62.2% of our visitors leaving inside 10 seconds against the holdout's 39.8%.
That is what routing cannot fix. If the page does not hold attention, choosing which page to serve is choosing between pages nobody reads.
Per-variant spread is larger than most people assume
One more number that argues for many variants rather than two. Six pages, same brand, same offer, same window:
| Page | Entrants | Conversions | CVR |
|---|---|---|---|
| teal-beacon-45 | 266 | 4 | 1.504% |
| amber-arbor-42 | 439 | 6 | 1.367% |
| early-delta-31 | 328 | 4 | 1.220% |
| azure-lagoon-33 | 447 | 5 | 1.119% |
| azure-thistle-81 | 287 | 2 | 0.697% |
| gentle-jasper-79 | 302 | 1 | 0.331% |
A 4.5× spread between best and worst. Conversion counts here are small — 1 to 6 — so individual rows are noisy and the extremes are the least reliable. But the spread is the point: if you had tested only two of these six, your read on "does page design matter for this offer" depends heavily on which two you happened to pick.
What to take from this
Aggregate ties can hide large contextual gains. If two pages tie on overall conversion rate, that is a reason to look for a segment flip, not a reason to conclude the pages are equivalent.
Demand the selection-versus-contextual split. Any uplift claim from a routing or personalisation system that does not decompose these is not evidence for the mechanism.
Ask which lift definition you are being shown. Best-page-versus-holdout and pooled-versus-holdout differ by tens of percentage points on the same data on the same day.
Routing cannot rescue a page nobody reads. Check dwell and scroll before you optimise allocation.
Lutiq predicts, for every paid click, which on-brand page to show — and measures lift against a live holdout. All figures above are pooled-versus-holdout unless stated. The measured campaign is an unnamed apparel brand. Figures are first-party unless attributed.
Open data
Machine-readable copies of the tables on this page (for citation and re-analysis):
- Toy contextual uplift example (JSON)
- Apparel campaign holdout summary 2026-07 (JSON)
- Research data index
The human article remains the canonical citation. JSON is for machines that prefer structured rows.