A/B Test Engine
Four things live together on this single page: a sample-size planner, a MDE minimiser, a significance calculator (frequentist + Bayesian) and a sequential peek-safe indicator. Paste one set of numbers and everything updates. Designed for product analysts running weekly experiments who are tired of five-tool pipelines.
Inputs
Posterior of difference (Beta-Binomial)
Sequential peek at this sample
Adjusts the naive z for planned interim looks using a simplified O'Brien-Fleming boundary (provided here for one planned interim peek). For continuous peeking, use an always-valid test like mSPRT.
Use cases
PM experiment design
Estimate sample required before launch, then re-check at the interim peek.
E-commerce promo A/B
Quantifies lift over 1 week of data and warns when subtle effects need 4× the sample.
Newsletter tests
P(T > C) is a more intuitive Bayesian readout than p-values when reporting to non-statisticians.
Method note
- Sample-size formula:
N/2 ≈ (zα/2+zβ)² · (p₁+p₂)(2−p₁−p₂) / (p₁−p₂)²(pooled-variance). - Frequentist test: two-prop z-test, two-sided.
- Bayesian: Beta(1,1) prior on each arm, post. = Beta(1+cv, 1+N−cv) — we sample 50,000 differences to estimate P(T > C).
- Sequential α: applied via Lan-DeMets approximation that gives a per-peek α multiplier.
When not to trust this
- Variance reduction / CUPED requires a covariate. We're not modelling that yet.
- Multiple variants inflate family-wise error; Bonferroni is a blunt tool but practical for early-stage teams.
- Don't read "P(T>C) = 99%" as "certain to ship" — priors matter, and we use a vague Beta(1,1).
FAQ
Why does the Bayesian number look so different from p-value?
P-value assumes the null is true and asks "how surprising is this data?" Bayesian asks "given this data, is T better than C?". The Bayesian number is closer to what stakeholders actually want.
Can I use mSPRT or always-valid?
Not directly here. As a rule of thumb: if you peek 5+ times pre-stopping, you must use an always-valid test.