Audit advanced Works with: Any AI

A/B Test Readout Quality Review

Table output Has example

Check conclusions drawn from an experiment.

The Open buttons send the prompt text as currently shown below to that site only when you click. Fill in the placeholders first.

Review an experiment readout. Hypothesis, variants and metrics: [TEST_DESIGN]. Visitors and conversions per variant: [RESULTS]. Dates and any events during the test: [DATES_AND_EVENTS]. The conclusion drawn: [CONCLUSION].

Check that the hypothesis and primary metric were set in advance, the sample size was planned, the test ran for full weeks, the split was balanced, guardrail metrics were checked, many metrics were not searched until one looked good, and the conclusion matches the evidence. Recompute the rates, the absolute and relative difference, and approximate intervals, showing the arithmetic.

Return a table of issues, a corrected conclusion in one of three forms (roll out, do not roll out, inconclusive), and a list of follow-up tests. Do not call a result proven from a small sample.

Add a worked recomputation. Variant A had 2,000 visitors and 90 conversions, 4.5 percent; variant B had 2,000 visitors and 112 conversions, 5.6 percent. Absolute difference 1.1 points; relative lift 24 percent. Approximate margin of error for A is 1.96 times the square root of 0.045 times 0.955 divided by 2,000 = 1.96 times 0.0046 = about 0.9 points, interval 3.6 to 5.4; for B about 1.0 points, interval 4.6 to 6.6. The intervals overlap slightly, so the result is suggestive but not conclusive; the readout claims B wins, which overstates. Check the other issues: the test ran 10 days and included a promotion on days 6 to 8, so repeat across full weeks; ten metrics were reviewed and the one that looked good was highlighted; a guardrail metric, refunds, was not checked. Write a corrected conclusion: Inconclusive: B shows a possible lift of about one point, but the intervals overlap and the test was disturbed by a promotion; extend for two more full weeks, keep one primary metric and check refunds. Add a follow-up test list with hypotheses and a rule for the planned sample size before starting. Include a template for readouts with context, hypothesis, results, interpretation, caveats and decision.

Fill in before use

Optional: type your details and the prompt above updates. Nothing is saved or sent.

Replace each bracketed placeholder with your own details, then paste into Any AI or another assistant.

Invented. A: 2,000 visitors, 90 conversions = 4.5 percent. B: 2,000 visitors, 112 conversions = 5.6 percent. Difference 1.1 points; relative lift = 1.1 / 4.5 = 24 percent.
Margin of error: A about 0.9 points (3.6 to 5.4); B about 1.0 (4.6 to 6.6). The intervals overlap.
Issues: test ran 10 days including a promotion on days 6 to 8; ten metrics were reviewed and the best one highlighted; refunds not checked.
Corrected conclusion: inconclusive; extend two full weeks with one primary metric and check refunds.

• audit