research
Ad creative embeddings cannot help when every click shares one creative set
Summary: We joined full ad-creative embeddings into a contextual landing-page allocation backtest covering 8,609 matched paid impressions. Every one of those impressions came from a single creative set. Under a paired ablation (real embedding vs indicator-only vs random vector vs seed re-runs), ad features added no reliable lift. If the ad side of the join never varies, the model cannot learn ad→page interactions — no matter how fancy the embedding.
It is tempting to believe that “add creative embeddings” will unlock predictive landing page allocation. Sometimes that is true. Often the data quietly refuses.
Setup
We re-ran a corrected walk-forward allocation backtest (same safeguards as Underpowered allocation evaluation) with ad-creative content joined as decision-time features: creative-set text, campaign context, asset filenames, and vision-derived descriptions embedded with MiniLM.
Join keys were hashed ad-set / campaign ids delivered by the ad platform macros (not raw platform ids in the clear). Coverage on the matched paid slice was 100%.
Open data: ad-feature-degeneracy-2026-07.json.
The structural fact
On the evaluation window (2026-07-08 → 2026-07-16):
| Quantity | Value |
|---|---|
| Matched impressions | 8,609 |
| Distinct creative sets in the matched set | 1 |
| Distinct ads in that set | 1 (two campaigns sharing the set) |
A feature that never varies carries no information beyond a match indicator the model already has through categorical ad ids.
Paired ablation (selection phase, argmax policy)
Six configs trained in one process on identical data, windows, propensities and bootstrap resamples. Paired delta vs baseline:
| Treatment | Paired Δ (abs CVR) | 95% CI | p(≤0) |
|---|---|---|---|
| Real ad embedding | +0.48pp | −0.31 … +1.18 | 0.31 |
| Indicator only | +0.15pp | −0.08 … +0.36 | 0.20 |
| Random vector, same width | +0.15pp | −0.58 … +0.68 | 0.32 |
| Same ads feature, different seed | +0.02pp | −0.43 … +0.47 | 0.49 |
| Baseline, different seed | −0.02pp | −0.04 … +0.10 | 0.81 |
On the full period, every paired delta sat within roughly ±0.07pp of zero.
Unpaired “ads model looks better on selection greedy” results inverted on the confirmation slice — the same instability pattern corrected methodology is designed to catch.
Interpretation
- Variation is a prerequisite for feature value. Rich creative content does not rescue a join where every impression shares one creative set.
- Always ablate. Indicator-only and random-vector controls expose seed noise dressed as creative intelligence.
- Operational implication: before investing in multi-modal ad embeddings for page allocation, report distinct creative sets per evaluation cell. If the count is 1, stop.
- Holdout still wins arguments. See Lutiq Holdout Protocol and What A/B tests cannot see.
Caveats
- One brand’s paid channel window; creative catalogue elsewhere was large (hundreds of sets) but not present in the matched impression join.
- Does not prove ad content never helps — only that it cannot help without variation in the logged traffic.
- Same conversion-rarity limits as the parent evaluation.