Testing Haven

MDE & Test Duration Calculator

What effect can you detect, and how long will it take?

Traffic and test parameters

%
%
%
%
Test type

To reliably detect a 5% relative lift on a 10.00% baseline, your test needs to run for about 12 weeks at 5,000 visitors per variant per week. That is a long test — consider a larger MDE, more traffic, or variance reduction (see the CUPED calculator).

Detectable effect by week

WeekVisitors / variantDetectable relative MDEDetectable rate
15,00017.4%11.74%
210,00012.2%11.22%
315,0009.9%10.99%
420,0008.6%10.86%
525,0007.6%10.76%
630,0007.0%10.70%
735,0006.4%10.64%
840,0006.0%10.60%
945,0005.7%10.57%
1050,0005.4%10.54%
1155,0005.1%10.51%
1260,0004.9%10.49%

Longer tests can detect smaller effects, but with diminishing returns — the MDE shrinks with the square root of sample size. If week 12 still cannot detect an effect worth shipping, the test is not worth running; see the CUPED calculator for ways to speed things up.

What this calculator does

This MDE and test duration calculator connects your real traffic to what is actually testable. Give it your weekly traffic, baseline conversion rate, and traffic split, and it shows the minimum detectable effect week by week — the smallest lift your test could reliably confirm after 1, 2, 3, … 12 weeks of runtime. Give it a target MDE, and it tells you how many weeks the test must run.

This is the planning question that comes before the sample size calculator: not "how many visitors do I need?" but "given the visitors I actually have, what experiments are even worth running?"

The math

Each week adds w⋅sw \cdot s visitors to the smaller arm, where ww is weekly traffic and ss is the smaller split share. The calculator then inverts the standard two-proportion power formula, solving for the effect size at which power reaches your target:

MDE(n)=min⁡{δ:  Φ ⁣(δ−z1−α/22pˉ(1−pˉ)np1(1−p1)+p2(1−p2)n)≥1−β}\text{MDE}(n) = \min \left\{ \delta : \; \Phi\!\left( \frac{\delta - z_{1-\alpha/2}\sqrt{\tfrac{2\bar{p}(1-\bar{p})}{n}}}{\sqrt{\tfrac{p_1(1-p_1)+p_2(1-p_2)}{n}}} \right) \geq 1 - \beta \right\}

solved numerically by bisection. Because nn appears under square roots, the MDE shrinks proportionally to 1/n1/\sqrt{n} — the source of the diminishing returns you see in the week-by-week table.

A worked example

Your checkout flow gets 20,000 visitors per week, converts at 10%, and you plan a 50/50 split (10,000 per arm per week) at 95% significance and 80% power. The table shows: after week 1 you can detect only a ~12% relative lift; by week 4 that improves to ~6%; by week 9 you reach ~4%. If the realistic upside of your change is a 5% lift, plan for about 6 weeks. If the realistic upside is 2%, this test is not runnable at your traffic — better to know that today.

When to use it

  • Roadmap triage: kill untestable experiment ideas before design and engineering time is spent.
  • Setting expectations: give stakeholders a runtime with a rationale, instead of "we'll see how it goes".
  • Choosing test populations: see how much faster the test finishes if you include more pages or segments in scope.

Common mistakes

  • Confusing traffic with test traffic. Only visitors who actually reach the tested experience count. If the change is on step 3 of checkout, use step-3 visitors, not site-wide sessions.
  • Stopping mid-week. Weekend and weekday visitors behave differently; partial weeks bias the sample. Round runtimes to whole weeks.
  • Extending a running test again and again. Deciding week by week whether to continue based on the current p-value is a form of peeking. Set the runtime up front — or use the sequential testing calculator, which is designed for continuous monitoring.
  • Ignoring variance reduction. If you have pre-experiment data on the same users, CUPED can cut these runtimes by 25–50% at typical correlations.

Frequently Asked Questions

What is a minimum detectable effect (MDE)?

The MDE is the smallest true effect your test can reliably detect at your chosen significance and power. If your test's MDE is an 8% relative lift and the variant actually improves conversion by 4%, the test will usually come back "not significant" — not because the effect is not real, but because your test cannot see effects that small at your traffic level.

How long should an A/B test run?

Long enough to collect the sample size required for your target MDE, and always in whole weeks to average out day-of-week patterns. In practice that is usually 2–6 weeks. Under one week risks unrepresentative traffic; beyond about 8 weeks you accumulate risks like cookie churn diluting your assignment, seasonal drift, and organizational impatience — at that point, test a bigger change instead.

Why does the detectable effect shrink so slowly week over week?

Because the MDE scales with one over the square root of the sample size. Doubling your runtime from 2 to 4 weeks only shrinks the detectable effect by about 29%. Going from week 8 to week 9 barely moves it. This is why "let it run a bit longer" is rarely the fix for an insensitive test — traffic, metric choice, and variance reduction (CUPED) are stronger levers.

Does an unequal traffic split help me test faster?

No — the opposite. Test sensitivity is limited by the smaller arm. A 90/10 split needs far more total traffic than 50/50 to reach the same MDE, because the 10% arm starves. Unequal splits are justified for risk containment (limiting exposure to a scary change), not for speed. This calculator uses the smaller arm to compute detectability, so you can see the cost directly.

What if the runtime for my target MDE is unacceptably long?

You have five options: (1) test a bolder change with a larger expected effect; (2) use a more sensitive upstream metric (clicks rather than purchases); (3) apply variance reduction like CUPED — a correlation of 0.6 cuts runtime by 36%; (4) include more traffic in the test population; (5) accept lower power or higher alpha, with eyes open. Running an 8%-MDE test and hoping for a 3% effect is not an option — it is a coin flip.