Testing Haven

CUPED Variance Reduction Calculator

Run faster tests with pre-experiment data

CUPED inputs

With a covariate correlation of ρ = 0.6, CUPED removes 36.0% of your metric's variance. Your test reaches the same power with 64% of the sample — 32,000 instead of 50,000 visitors per variant, saving 18,000 visitors per variant.

What this means

  • Variance reduction: 36.0% (= ρ²)
  • Effective sample size multiplier: 1.56× — your data works as hard as a 78,125-visitor test without CUPED
  • Test duration multiplier: 0.64× — a 4-week test finishes in about 2.6 weeks

Pair this with the sample size calculator — multiply its answer by 0.64 to get your CUPED-adjusted requirement.

What this calculator does

This CUPED calculator quantifies how much faster your experiments could run if you used pre-experiment data to cancel noise. Enter the correlation ρ between your metric and a pre-experiment covariate (and optionally your planned sample size), and it returns the variance reduction, the effective sample size multiplier, and the concrete savings: how many visitors — and how many weeks — CUPED would shave off your test.

Variance reduction is the most underused lever in A/B testing: teams agonize over traffic they cannot increase while ignoring variance they could remove for free.

The math

Let YY be your metric and XX a covariate measured before the experiment. CUPED replaces YY with the adjusted metric:

Y′=Y−θ (X−E[X]),θ=Cov(X,Y)Var(X)Y' = Y - \theta\,(X - \mathbb{E}[X]), \qquad \theta = \frac{\mathrm{Cov}(X, Y)}{\mathrm{Var}(X)}

Because XX predates assignment, it is independent of treatment, so the expected difference between arms is untouched — but the variance becomes:

Var(Y′)=Var(Y) (1−ρ2)\mathrm{Var}(Y') = \mathrm{Var}(Y)\,(1 - \rho^2)

where ρ\rho is the correlation between XX and YY. Since required sample size is proportional to variance, a test that needed nn visitors needs only n(1−ρ2)n(1 - \rho^2) after CUPED.

A worked example

Your revenue test needs 200,000 users per arm (from the sample size calculator) — ten weeks at your traffic. Last-4-weeks revenue per user correlates with in-experiment revenue at ρ = 0.65 for your returning users. CUPED removes ρ² = 42% of the variance, so the requirement drops to 200,000 × 0.58 = 116,000 users per arm — about 5.8 weeks instead of 10. Same decision quality, four weeks sooner, using data you already have.

When to use it

  • Products with returning users and metrics that persist per user (revenue, engagement, retention proxies).
  • High-variance metrics like revenue per visitor, where tests are painfully long — combine with the RPV calculator.
  • When the MDE calculator says your target effect needs more weeks than stakeholders will tolerate.

Common mistakes

  • Using a covariate measured during the experiment. This is the one fatal error: anything the treatment can influence will bias the treatment effect estimate itself, not just the variance. The covariate window must end at assignment.
  • Expecting magic for new-user experiments. No pre-period data means ρ ≈ 0 and no gain. CUPED complements, not replaces, traffic planning for acquisition tests.
  • Estimating θ separately per arm. Estimate it once on pooled data; per- arm estimates reintroduce bias.
  • Shopping among covariates after seeing results. Trying five covariates and reporting the most flattering adjustment is p-hacking in a variance-reduction costume. Pre-register the covariate.
  • Overstating ρ. Measure the correlation on recent historical data for the exact population and window you will use — do not borrow a number from a blog post (including this one).

Frequently Asked Questions

What is CUPED in one paragraph?

CUPED (Controlled-experiment Using Pre-Experiment Data) is a variance-reduction technique from Microsoft's experimentation team. It subtracts the predictable part of each user's metric — predicted from their own pre-experiment behavior — so the experiment only has to detect the treatment effect against the residual, unpredictable noise. The treatment effect estimate is unchanged and unbiased; its noise shrinks by a factor of 1 − ρ², where ρ is the correlation between the covariate and the metric.

What covariate should I use?

The best covariate is usually the same metric measured on the same user before the experiment started: pre-period conversion for a conversion test, pre-period revenue for a revenue test, measured over a window of similar length (1–4 weeks). It must be measured entirely before assignment — a covariate influenced by the treatment does not just weaken CUPED, it biases the result.

What correlation values are realistic?

For engagement and revenue metrics on returning users, ρ between 0.5 and 0.8 is common (users who visited a lot last week visit a lot this week). For purchase conversion, 0.3–0.6 is typical. For new users, ρ is near zero because there is no pre-period data — which is why CUPED helps most on products with repeat usage, and why the standard practice is to apply it to returning users and leave new users unadjusted.

Does CUPED change what my experiment measures?

No. The CUPED-adjusted estimator targets exactly the same treatment effect and is unbiased regardless of the covariate (as long as the covariate is pre-experiment). CUPED only removes variance — it is the statistical equivalent of a cleaner signal path, not a different measurement.

How do I actually implement CUPED?

For each user, compute the adjusted metric Y′ = Y − θ(X − mean(X)), where X is the pre-experiment covariate and θ = cov(X, Y) / var(X), estimated on the pooled experiment data. Then run your usual test on Y′ instead of Y. It is a few lines of SQL or pandas. Estimate θ once across both arms (not per arm), and pre-register the covariate choice to avoid fishing.