Testing Haven

Revenue per Visitor Calculator

Test revenue and other continuous metrics

Raw values per visitor

Test(Mann-Whitney is robust to heavy-tailed revenue outliers)
Test type

Paste at least two numeric values per group to run the test.

What this calculator does

This calculator runs statistical tests on continuous metrics — revenue per visitor, average order value, items per cart, time on task — the metrics conversion calculators cannot touch. Paste the raw per-visitor values for control and variant (straight from a spreadsheet or SQL export; commas, spaces, or newlines all work), choose Welch's t-test or the Mann-Whitney U test, and get the means, the difference with a confidence interval, a p-value, and a plain-English verdict.

The formulas

Welch's t-test compares means without assuming equal variances:

t=xˉ2−xˉ1s12n1+s22n2,ν≈(s12n1+s22n2)2(s12/n1)2n1−1+(s22/n2)2n2−1t = \frac{\bar{x}_2 - \bar{x}_1}{\sqrt{\dfrac{s_1^2}{n_1} + \dfrac{s_2^2}{n_2}}}, \qquad \nu \approx \frac{\left( \tfrac{s_1^2}{n_1} + \tfrac{s_2^2}{n_2} \right)^2}{\tfrac{(s_1^2/n_1)^2}{n_1 - 1} + \tfrac{(s_2^2/n_2)^2}{n_2 - 1}}

where s12,s22s_1^2, s_2^2 are the sample variances and ν\nu is the Welch–Satterthwaite degrees of freedom. Mann-Whitney U instead ranks all values together and asks whether one group's ranks are systematically higher:

U=R1−n1(n1+1)2U = R_1 - \frac{n_1(n_1+1)}{2}

with R1R_1 the rank sum of group 1; the calculator applies the normal approximation with tie and continuity corrections.

A worked example

You test a free-shipping threshold. Control: 4,000 visitors averaging $3.10 per visitor. Variant: 4,000 visitors averaging $3.42 (+10.3%). Revenue is spiky — standard deviations near $18 in both arms. Welch's test gives t ≈ 0.79 and p ≈ 0.43: despite the attractive +10% headline, this difference is well within noise at this sample size. Mann-Whitney agrees (p ≈ 0.51). The verdict block says exactly that — and the honest readout is "we need roughly 4x this traffic for a lift this size", not "variant wins by 10%".

When to use it

  • Pricing, shipping, bundling, and upsell tests, where the money moves through order size rather than conversion.
  • Any test where the honest primary metric is revenue per visitor, not clicks — pair it with the significance calculator on the conversion metric to see the whole picture.
  • Non-monetary continuous metrics: session duration, items viewed, scroll depth.

Common mistakes

  • Excluding zeros — that silently converts revenue-per-visitor into average order value, a different metric with different dynamics.
  • Letting whales decide the test. One $8,000 order in either arm can dominate everything. Decide a winsorization cap before the test, and use Mann-Whitney as a robustness check.
  • Testing the mean but reporting the median (or vice versa). Mann-Whitney significance does not certify a mean lift — be precise about which claim you ship.
  • Assuming session-level values are independent when the same user appears in many rows. Aggregate to one value per randomized unit (usually per user) before pasting.
  • Underpowering. Revenue tests need dramatically more data than conversion tests; check feasibility with the MDE calculator and consider CUPED to claw back variance.

Frequently Asked Questions

Why can't I use a normal conversion-rate calculator for revenue?

Conversion calculators assume each visitor is a yes/no outcome. Revenue per visitor is a continuous amount — mostly zeros, plus purchases of wildly different sizes — so the right tools are two-sample tests on means or distributions: Welch's t-test or the Mann-Whitney U test. Reducing revenue to "converted or not" throws away exactly the information (order values) a revenue test exists to measure.

Should I include visitors who spent nothing?

Yes, include every visitor with a zero. Revenue per visitor is defined over all visitors; dropping non-buyers turns the metric into average order value and silently changes the question from "does this make more money per visitor?" to "do buyers spend more?" — a different quantity that can move in the opposite direction.

Welch's t-test or Mann-Whitney — which should I use?

Welch's t-test compares means — the business quantity — and is reliable at e-commerce sample sizes despite skewed data, thanks to the central limit theorem. Mann-Whitney asks whether one group's values tend to be larger, and is far more robust to extreme outliers, but it does not directly test the mean. A sensible workflow: use Welch as primary; if a handful of whale orders flips your Welch conclusion, let Mann-Whitney arbitrate whether the effect is broad or one lucky order.

Why does Welch's test not assume equal variances?

The classic Student t-test pools the two groups' variances, which biases results when variances differ — and in revenue tests they almost always differ, since a variant that changes purchase behavior changes revenue variance too. Welch's version estimates each variance separately and adjusts the degrees of freedom (Welch–Satterthwaite), which is why it is the recommended default in modern practice.

How many visitors do I need for a revenue test?

Typically many more than a conversion test on the same page, because revenue variance is dominated by a small number of large orders. As a rule of thumb, expect 2–5x the sample of a conversion test for the same relative MDE. You can shrink this substantially by winsorizing extreme values (with a pre-registered cap) or using CUPED with pre-experiment revenue — see the CUPED calculator.