Testing Haven

Sequential Testing Calculator

Peek at results without inflating false positives

Current test data (peek any time)

Enter the current visitors and conversions for both groups. Unlike a fixed-horizon test, you can check this as often as you like.

Why peeking breaks fixed-horizon tests

What this calculator does

This sequential testing calculator lets you monitor an A/B test continuously and stop the moment the evidence is strong — without the false-positive inflation that makes "peeking" at classical tests infamous. Enter the current visitors and conversions for control and variant, as often as you like, and it returns an always-valid p-value and confidence interval computed with the mixture sequential probability ratio test (mSPRT), the method behind Optimizely's Stats Engine and the "peeking at A/B tests" literature.

Almost every team peeks — the honest fix is not more discipline, it is statistics designed for peeking.

How the mSPRT works

The test tracks a likelihood ratio comparing "no effect" against a mixture of possible effects θ∼N(0,τ2)\theta \sim N(0, \tau^2). For an observed rate difference θ^\hat{\theta} with estimated variance VV, the mixture likelihood ratio has a closed form:

Λ=VV+τ2  exp⁡ ⁣(θ^2 τ22V(V+τ2))\Lambda = \sqrt{\frac{V}{V + \tau^2}}\; \exp\!\left( \frac{\hat{\theta}^2\, \tau^2}{2V(V + \tau^2)} \right)

Under the null hypothesis, Λ\Lambda is a martingale — its expected value never grows, no matter how long you watch it. That is the mathematical reason you may peek freely: the always-valid p-value p=min⁡(1,1/Λ)p = \min(1, 1/\Lambda) can only be pushed below α\alpha by genuine evidence, with probability at most α\alpha over the entire lifetime of the test.

A worked example

Day 6 of a test: control 8,000 visitors / 800 conversions (10.0%), variant 8,000 / 920 (11.5%). A naive fixed-horizon z-test gives p ≈ 0.002 — but you have been checking every day, so that number no longer means what it claims. The mSPRT gives an always-valid p ≈ 0.02. Still below α = 0.05, so you can stop and ship — and this conclusion is honest about the daily monitoring. If the always-valid p had been 0.11, the right move would be to keep collecting, knowing tomorrow's check costs nothing statistically.

When to use it

  • Any test whose results appear on a live dashboard that people actually look at.
  • Tests where shipping early has real value (fixing a losing variant fast).
  • Auditing past "wins" that were declared the day they crossed p < 0.05.
  • When you cannot pre-commit to a sample size — though it is still worth checking the MDE calculator to confirm the test is feasible at all.

Common mistakes

  • Applying sequential thresholds to a fixed-horizon p-value. The p-value on your analytics dashboard is almost certainly fixed-horizon; treating it as safe to monitor is precisely the peeking problem. Use this calculator's p-value instead.
  • Stopping on the naive p-value and "confirming" with mSPRT later. Decide with the always-valid number; the naive one is for curiosity only.
  • Skipping data-quality checks because the stats are fancy. mSPRT assumes valid randomization — run the SRM checker alongside it.
  • Restarting the test after a near-miss. Killing a test at p = 0.06 and relaunching it "fresh" is peeking with extra steps. The martingale only protects you if the test keeps its history. Read more in our peeking guide.

Frequently Asked Questions

What is an always-valid p-value?

A p-value you can look at whenever you like — after every visitor, every hour, every day — and act on the moment it crosses your threshold, without inflating your false positive rate. Classical p-values are only valid at one pre-planned sample size; always-valid p-values from the mSPRT hold simultaneously at every sample size, which is exactly the guarantee continuous monitoring needs.

Why exactly does peeking break a normal A/B test?

A fixed-horizon p-value is a random number that wobbles as data arrives, and under the null hypothesis it dips below 0.05 by luck about 5% of the time at any single look. If you check 20 times and stop at the first dip, you give yourself 20 chances at that 5% event — the effective false positive rate can climb past 20–30%. The simulation chart on this page shows it happening.

What is the catch compared to a fixed-horizon test?

Sequential tests pay for their flexibility with efficiency: to keep the error guarantee at every possible stopping point, the evidence bar is higher, so a true effect of a given size needs somewhat more data to reach significance than a perfectly-executed fixed-horizon test would need. In practice the trade is worth it — fixed-horizon tests are rarely perfectly executed, and stopping early on strong effects usually recovers the difference.

What is the tau (τ) parameter?

The mSPRT needs a prior guess about the scale of plausible effects, expressed as the standard deviation τ of a normal distribution centered at zero. This calculator defaults τ to 10% of your pooled conversion rate, a reasonable scale for typical A/B lifts. The test stays valid for any τ — a mismatched τ only costs detection speed, never correctness.

Can I use this on a test I already peeked at?

Yes — that is one of its best uses. If you peeked at a fixed-horizon test and it "hit significance", recompute it here. The always-valid p-value tells you whether the evidence is strong enough to survive the peeking you actually did. If mSPRT says significant, you are safe; if not, the fixed-horizon significance was likely a peeking artifact.

When should I still prefer a fixed-horizon test?

When you genuinely will not peek (results are hidden until a scheduled readout), when you need maximum sensitivity per visitor on scarce traffic, or when the runtime is fixed for external reasons anyway (a two-week campaign). In those cases the classical test plus a pre-computed sample size is slightly more efficient. For everything else — dashboards that update live, stakeholders who ask daily — sequential is safer.