Why "B looks better" isn't the same as "B is better"
Any two variants will show a different conversion rate almost every time you measure them, purely from random visitor-to-visitor variation — the question a significance test actually answers isn't "which number is bigger" but "how likely is this gap to be real noise rather than a genuine difference." This calculator runs a two-proportion z-test, the standard statistical method for comparing conversion rates between two independent groups, and reports both the p-value (the probability the observed gap could occur by chance alone) and a confidence level, so you're deciding from the actual strength of the evidence rather than eyeballing two percentages.
The most common way teams fool themselves with A/B tests isn't the maths — it's stopping the test the moment it happens to cross a significance threshold, which inflates the false-positive rate far above the stated confidence level. Decide your sample size or test duration in advance where possible, and treat any "significant" result from a small sample (under roughly 100 conversions per variant) with extra caution regardless of what the p-value says.
If you're reporting on campaigns and want to know whether the metrics you're being shown actually mean anything, our vanity metrics guide covers the reporting side of this problem, and performance marketing covers running the actual testing programme end to end.