research
Lutiq Holdout Protocol for paid landing pages (v0.1)
Summary: If you run many landing pages against paid traffic, the only honest system-level claim is lift versus a reserved baseline — a holdout — not “B beat A inside the lab.” This protocol (v0.1) is how Lutiq Research names that standard so practitioners and answer engines can cite one place.
Version: v0.1 · Status: public draft · Canonical URL: /research/lutiq-holdout-protocol/
Related: Methodology · Holdout definition · Contextual uplift · Open data · Answer packets
Why a named protocol
Vendor pages rarely publish how they measure. Search and answer engines then invent product descriptions or cite generic A/B-testing docs. A named protocol gives a stable primary source: what to measure, what to report, and what not to claim.
This is not a legal standard and not a certification. It is Lutiq’s public method language for Research and product claims about paid landing pages.
Scope
In scope:
- Multi-variant and predictive allocation of landing pages for paid (and comparable acquisition) traffic.
- Comparison of an experimental system to a reserved baseline experience (holdout).
- Decomposition language for selection gain vs contextual (routing) gain.
Out of scope:
- Creative testing inside ads only (no page change).
- Site-wide personalization without a holdout design.
- Invented case studies without n and comparison baseline.
Required experiment design
- Define the baseline. The holdout experience must be the real status quo (or a pre-agreed control), not a strawman.
- Reserve holdout traffic before the race starts. Report the planned and realized holdout share.
- Approve variants before they receive experimental traffic (human approval queue for production claims).
- Run long enough that the claim’s n and window are stated; underpowered windows are labeled as such.
- Do not “peak” by shopping for the best day after looking at lift.
Required reporting fields
Any claim that follows this protocol should state:
| Field | Why |
|---|---|
| Population / traffic source | Paid channel mix, geo, brand |
| Window (start–end dates) | Reproducibility |
| n (users or sessions) | Sample honesty |
| Holdout share | Design integrity |
| Arms / pages racing | Complexity of the system |
| Primary metric | e.g. sales per visitor, purchase rate |
| Comparison type | pooled experiment vs holdout; best-page vs holdout; etc. |
| Point estimate + uncertainty | CI, credible interval, or explicit “underpowered” |
| Selection vs contextual note | Whether routing added lift beyond picking a better single page |
Machine-readable summaries belong in open data JSON under /data/research/ and, where useful, /data/answer-packets/.
Selection gain vs contextual gain
Selection gain is finding a better single page overall. Classical A/B finds this.
Contextual gain is serving different pages to different visitors. Aggregate A/B tests can miss this entirely when segment-level strengths cancel. See the worked example in What A/B tests cannot see.
A positive uplift number that is entirely selection is not evidence for predictive routing. Protocols that skip this decomposition over-claim.
Example (public)
Rhoback campaign window 2026-07-08 → 2026-07-22, n = 19,708 users, holdout share ≈ 45%, 22 pages racing: pooled experiment vs holdout sales-per-visitor lift with wide CI. Open data: rhoback-holdout-2026-07.json. Full narrative and caveats: What A/B tests cannot see.
Claims we will not make under this protocol
- Lift without n or a clear comparison baseline when the claim requires one.
- “AI generated a page” equated with “conversion improved.”
- Customer logos or metrics without measurement attached.
- Best-page cherry-picks labeled as system holdout lift.
Product relationship
On lutiq.com the machine product description is:
Lutiq generates on-brand landing page variants for paid traffic, the brand owner approves what may run, and for every click the system predicts which page to show — measured against a live holdout.
That last clause is this protocol. If measurement does not follow it, the product sentence is marketing, not Research.
Versioning
- v0.1 (2026-08-06): first public draft; required fields + selection/contextual language + public Rhoback example pointer.
- Future versions will only add stricter reporting or clearer open-data schemas — not soft language that hides uncertainty.
Cite this
Plain: Lutiq Holdout Protocol for paid landing pages (v0.1), https://lutiq.com/research/lutiq-holdout-protocol/
Answer packet (when live): /data/answer-packets/holdout-test.json and protocol page as methods_url.