Prove
Experiments
Most promotions are unmeasurable by construction: they run everywhere at once, so there is nothing to compare against. Both the holdout and the metric are decided before the promotion, or not at all.
Minimum detectable effect at 95% confidence and 80% power. The noise that matters is the variability of the GAP between the site and its synthetic control, not the site's own variability — a volatile site is still cheap to measure if its control is volatile in the same way.
How long the window has to be
Detectability improves with the square root of the window, so doubling a promotion’s length buys about a 30% smaller detectable effect — not half. That is why a three-day promotion is usually not worth measuring: at 10.7% the effect would have to be implausibly large to show up at all.
Every site, at 14 days
| site | daily mean | gap sd | MDE | donors |
|---|---|---|---|---|
| DXB-DEI | 759 | 94.1 | 9.28% | DXB-DTN DXB-MAR DXB-MOE |
| DXB-DTN | 1,286 | 79.5 | 4.63% | DXB-DEI DXB-MAR DXB-MOE |
| DXB-MAR | 1,327 | 87.8 | 4.95% | DXB-DEI DXB-DTN DXB-MOE |
| DXB-MOE | 1,600 | 122.5 | 5.73% | DXB-DEI DXB-DTN DXB-MAR |
Deira has the smallest absolute traffic and the largest detectable effect, because its gap to the synthetic control is noisy relative to its own size. It is the site where a promotion is hardest to prove worked.
Pre-registration
Decided BEFORE the promotion, or not at all. If a promotion is not expected to clear the MDE for its window, it should run longer or not be measured — running it and reporting whatever number comes out is what this page exists to prevent.