Create advanced Works with: Any AI

A/B Test Results Interpretation

Table output Has example

Read results without fooling yourself.

The Open buttons send the prompt text as currently shown below to that site only when you click. Fill in the placeholders first.

Interpret A/B test results. Variant A and B visitors and conversions: [AB_DATA]. Planned sample size and duration: [PLAN]. Other metrics: [OTHER_METRICS].

Compute: conversion rates, absolute and relative lift, a two-proportion z-test (pooled proportion p = (x1 + x2)/(n1 + n2); standard error = sqrt(p(1-p)(1/n1 + 1/n2)); z = (p2 - p1)/SE), an approximate p-value and a 95 percent confidence interval for the difference, with arithmetic shown. State clearly whether results are consistent with no difference.

Check validity: sample ratio mismatch, run length, novelty effects, segments (caution against fishing), guardrails. Produce a decision recommendation based on the pre-agreed rule and the practical significance (is the lift worth the cost?). Write a summary for stakeholders (120 words) in plain language. Remind me that I should verify calculations with a statistical tool. Do not draw conclusions the data cannot support.

Report the effect size with its interval, state plainly whether it meets the pre-agreed rule, and add a final line: ship, extend, or stop, with the reason. If the sample is below what was planned, say the result is underpowered.

Fill in before use

Optional: type your details and the prompt above updates. Nothing is saved or sent.

Replace each bracketed placeholder with your own details, then paste into Any AI or another assistant.

Variant A: 4,000 visitors, 200 conversions (5.0 percent). Variant B: 4,000 visitors, 236 conversions (5.9 percent).
Lift = 0.9 points absolute, 18 percent relative.
Pooled p = 436 / 8,000 = 0.0545. SE = square root of (0.0545 x 0.9455 x (1/4,000 + 1/4,000)) = 0.00508.
z = 0.009 / 0.00508 = 1.77. Two-sided p-value about 0.077.
95 percent interval for the difference: 0.009 +/- 1.96 x 0.0051 = -0.001 to 0.019.
Conclusion: consistent with no change or a modest gain. The pre-set rule required p under 0.05, so do not ship on this evidence; extend the test or accept the uncertainty knowingly. Verify with a statistics tool.

• ab-test