Create intermediate Works with: Any AI

Multiple Comparisons and False Positives Explainer

Table output Has example

Understand why many tests produce spurious wins.

The Open buttons send the prompt text as currently shown below to that site only when you click. Fill in the placeholders first.

Explain multiple comparisons using my situation. Analysis I am running: [ANALYSIS]. Number of tests or comparisons: [NUMBER_OF_TESTS]. Significance level I use: [ALPHA].

Explain in plain language why testing many things inflates false positives, and calculate the chance of at least one false positive for my number of tests with the formula shown. Compare corrections (Bonferroni, Holm, false discovery rate) in a table, with when each is used and its cost in power. Apply two of them to a made-up set of ten p-values so I can see the effect. Recommend practical approaches for my case, such as pre-registering one primary comparison and treating others as exploratory. Include wording for reporting honestly.

Add a short simulation I can run in Python that draws many comparisons from random data and counts how many appear significant, with the code and the expected pattern. Include a plain-language paragraph on how to describe exploratory findings in a report without overclaiming.

Fill in before use

Optional: type your details and the prompt above updates. Nothing is saved or sent.

Replace each bracketed placeholder with your own details, then paste into Any AI or another assistant.

Your case: 20 tests at alpha 0.05. Chance of at least one false positive = 1 - 0.95^20 = 0.64.
Corrections
Bonferroni: use 0.05/20 = 0.0025 per test; simple, loses power.
Holm: step-down, slightly more powerful.
FDR (Benjamini-Hochberg): controls the share of false discoveries.
Made-up p-values: 0.001, 0.004, 0.012, 0.03, 0.04, ...
At Bonferroni only the first two pass; with FDR at 5% three pass.
Practical: pre-register one primary comparison; treat the rest as exploratory; say how many tests you ran.

• stats