glossary
Holdout test: definition for landing pages and conversion systems
A holdout test reserves a portion of traffic for a baseline experience while other traffic receives a new treatment (a new page, a multi-variant pool, or a prediction router). You compare outcomes between holdout and treatment to estimate true lift versus the status quo.
It answers a different question than a classic A/B test:
| Design | Question |
|---|---|
| A/B (A vs B) | Which of these two is better relative to each other? |
| Multi-variant + prediction | Which page should this click see, given evidence so far? |
| Holdout | Is the new system better than the baseline we would have run anyway? |
Why paid traffic needs holdouts
Paid creatives rotate, auctions move, and offers change. Variant B can beat variant A inside an experiment while absolute ROAS falls versus last month’s baseline. A holdout keeps that baseline honest.
Design notes
- Define the baseline clearly (old PDP, campaign URL, homepage).
- Size the holdout so lift is estimable without starving the learning system.
- Do not “quietly improve” the baseline mid-test.
- Match attribution windows across arms.
- Segment by major channel when reporting.
Related
- Short guide: What is a holdout test?
- Contextual uplift example: What A/B tests cannot see
- Product use of holdouts: About Lutiq