Skip to main content
Clicks and impressions show that visitors noticed a message, not that it sold anything: many of those visitors would have bought anyway. So each message is measured against a holdout.

​
The holdout

Some visitors who would see the message are picked at random and never shown it. Everything else about them is the same: they match the same audience and the same trigger fires for them. Comparing what they spend with what visitors who saw the message spend shows how much the message itself brings in.
  • 10% of visitors are held out by default. Change this on the message’s Measure tab, from 0% to 50%. A larger holdout gives a clear answer sooner, but fewer visitors see the message.
  • Each visitor stays in the same group for a message, every time.
  • An order counts if it’s placed within 7 days of the message opening, or of when it would have opened for a held-out visitor.
  • Changing a message’s content or holdout starts a new set of results, so results never mix two versions.
With a 0% holdout, the message still shows its impressions, clicks and closes, but there’s nothing to compare with, so it has no lift.

​
Reading the lift

The list of messages, and the Measure tab, show the change in revenue per visitor, with a 95% range:
+12% revenue per visitor
95% range +4% to +21%
  • +12% is the best estimate: visitors shown the message spent 12% more on average than held-out visitors.
  • The 95% range is how sure that estimate is. The true effect is very likely between +4% and +21%.
How to read the range: The Measure tab also shows the change in conversion rate (the share of visitors who ordered) with its own range, how many visitors are in each group with each group’s conversion rate and revenue per visitor (in your store’s currency, with orders in other currencies converted), and what shown visitors did: impressions, clicks and closes.

​
Not enough data yet

A lift needs at least 100 visitors and 10 orders in each group. Until then, the message shows Not enough data yet, and the Measure tab says what’s still needed, such as “40 more orders from held-out visitors”. With a 10% holdout, the held-out group usually takes longest.