You Can’t Call It Incremental If You Can’t Measure It

Incrementality is becoming one of advertising’s most overused claims. Without a true holdout and a disciplined test, reported performance can’t tell brands what would have happened anyway.

Choosing a new DSP partner should really come down to one question: Does it bring genuinely different intelligence, or is it just another place to spend the same budget? That question only has a real answer if incrementality is measured, not just claimed. 

“Incrementality” should become the standard advertisers reach for when judging a new platform, a new algorithm, or a new audience strategy. It’s the right standard, but it is also becoming a word people use without a method behind it. A platform review that concludes a partner is “driving incremental results” based on its own reported conversions hasn’t tested incrementality at all. It has repeated the platform’s homework.

Before incrementality can become the standard, it needs a measurement framework. That requires a different kind of test, and most media organizations are not yet set up to run it.

Reported performance isn’t proof of incrementality

Every platform can report CPA or ROAS alongside a conversion count. None of those numbers say what would have happened without the campaign. A platform that reports strong performance may simply be claiming credit for people who were going to convert anyway, or who another partner in the stack was already reaching. Two DSPs can both report attractive results while splitting the same pool of likely buyers, and the reporting from each will look identical to genuine growth.

This is a structural issue rather than a data-quality problem. The reporting may be accurate on its own terms while remaining unable to establish causality.

The holdout is the only honest comparison

A real incrementality test compares exposed and unexposed groups that are otherwise identical. It then measures the gap in outcomes between them. 

In digital media, that typically means using a public service ad or segment test, with a portion of the eligible audience withheld from the campaign entirely. Advertisers can also use a geo-based holdout, with matched markets serving as test and control.

Give the test enough time to mean something

A holdout that runs for a week captures noise more than it captures signal, particularly for products with a longer consideration cycle. Advertisers should plan for enough flight time to accumulate a meaningful sample in both groups, plus additional time after spend ends for delayed conversions to appear.

Ending a test early because the topline numbers look promising is one of the most common ways an incrementality read gets corrupted. The lift that shows up in week two doesn’t always hold in week four, and the only way to know is to let the test run its full course.

Isolate what’s actually being tested

A platform test answers a different question from an audience or creative test. Collapsing them into one read produces a clear answer to none of them. If a new DSP is being evaluated, the audience strategy and creative should stay constant across test and control so that any lift can be attributed to the platform’s decisioning, not to a different targeting approach running underneath it.

The same logic applies to evaluating a new audience or a new algorithm. Whatever variable is under review should be the only one that changes. Everything else needs to be held still long enough to isolate its effect.

What a credible incrementality test requires

Before launching a test that will inform a real budget decision, advertisers should be able to answer the following questions:

• Is there a true holdout, not just a reporting comparison? A control group that isn’t exposed to the campaign is the only way to see the counterfactual.
• Is the sample size large enough to detect a meaningful lift? Small holdouts produce results that look directional but aren’t statistically reliable.
• Is the flight length matched to the conversion cycle? Tests ended before lag time has played out will understate or misstate the result.
• Is every other variable held constant? All test conditions beyond the variable under review should stay fixed for the duration of the test.
• Is there a predetermined threshold for what counts as a win? Deciding what “enough lift” looks like after seeing the results invites the test to confirm whatever the organization already wanted to believe.

Incrementality is a discipline

Incrementality testing takes work, and directional reads may be sufficient for lower-stakes decisions. For decisions that materially affect the media stack or budget allocation, however, a claim of incrementality is only as strong as the test behind it.

Advertisers get real value from their media stack by looking beyond the highest reported ROAS. They build the discipline to ask what would have happened anyway and trust a platform’s contribution only once a proper test has answered that question.