A/B Test Design Document
Design an experiment before running it.
The Open buttons send the prompt text as currently shown below to that site only when you click. Fill in the placeholders first.
Design an A/B test. Hypothesis and change: [HYPOTHESIS_CHANGE]. Primary metric and baseline: [PRIMARY_METRIC_BASELINE]. Traffic available per week: [TRAFFIC]. Risks: [RISKS]. Write the test document: hypothesis in the form "if we do X, then Y will change because Z"; primary, secondary, and guardrail metrics; unit of randomisation; sample size calculation using the formula and my inputs with the assumed minimum detectable effect and significance and power levels I choose [CHOICES], showing the arithmetic; duration; exclusions; and a pre-registered analysis plan with a rule against peeking. Add a decision table: what we do for win, loss, no result. Explain novelty effects and multiple comparisons. Do not claim results; the document is for planning. Add a short worked example of the sample size arithmetic with invented numbers, clearly labelled as an illustration, and a list of the mistakes that most often invalidate tests, such as changing the test midway or stopping when the result looks good. Include a section on how to communicate the result honestly, including inconclusive outcomes. Add a worked sample size calculation with invented inputs: baseline conversion 4.0 percent, minimum detectable effect of 0.5 percentage points (relative lift 12.5 percent), significance 5 percent two-sided, power 80 percent. Show the formula for a two-proportion test, the substitution step by step, and the result per variant, then convert to days using weekly traffic of 40,000. Label every figure an illustration and tell me to verify with my own tool. Add a pre-launch checklist (tracking verified in both variants, assignment balanced, no overlapping tests on the same users, guardrail metrics defined) and a post-test template: result, interval, guardrails, segments looked at and why, decision, and learning for the log. Close with a one-paragraph description of what would make the test invalid and need restarting, such as a tracking bug or a major release during the test, and who decides.
Fill in before use
Optional: type your details and the prompt above updates. Nothing is saved or sent.
Replace each bracketed placeholder with your own details, then paste into Any AI or another assistant.