A/B Test Calculator

An A/B test randomly assigns comparable visitors to a control experience and an alternative experience, then measures a predefined outcome such as a purchase, signup, click, or completed task. This calculator compares two binary conversion rates using a pooled two-proportion z-test. It reports each rate, relative uplift, the z-score, a two-sided p-value, and whether the result crosses the conventional 0.05 significance threshold. For each variant, conversion rate equals conversions divided by visitors. Under the null hypothesis that both variants have the same underlying rate, the calculator pools conversions across both samples. It uses that pooled rate to estimate the standard error of the difference, then divides the observed rate difference by the standard error to obtain a z-score. The two-sided p-value measures how surprising a difference at least this large would be if the null hypothesis were true. A p-value below 0.05 is commonly described as statistically significant at the 95 percent confidence level. It does not mean there is a 95 percent probability that the test variant is better, nor does it measure the business value of the change. Relative uplift compares the rate difference with the control rate, while absolute uplift is simply rate B minus rate A. A small but statistically reliable lift may have substantial value at scale, and a large observed lift may remain uncertain when the samples are small. Valid inference depends on experimental design. Assign users randomly, keep assignment stable, avoid counting the same independent unit multiple times, choose the primary metric before inspecting results, and run the experiment long enough to capture normal cycles. Stopping whenever significance first appears inflates false-positive risk. Multiple metrics, repeated peeking, and many variant comparisons also require adjusted analysis or a planned sequential method. The normal approximation works best when each variant has enough expected conversions and non-conversions. Very sparse outcomes may require Fisher's exact test or another exact method. Statistical significance also does not protect against instrumentation errors, novelty effects, sample-ratio mismatch, bot traffic, or changes that occurred during the test. Use this result with confidence intervals, guardrail metrics, practical effect thresholds, and a review of experiment quality before making a rollout decision.