Create intermediate Works with: Any AI

Consistency Check Across Repeated Runs

Table output Has example

Measure how much outputs vary.

The Open buttons send the prompt text as currently shown below to that site only when you click. Fill in the placeholders first.

Design a consistency test for a prompt. The prompt: [PASTE_PROMPT]. A set of 5 typical inputs: [SAMPLE_INPUTS]. What must stay the same across runs (decisions, numbers, format, labels): [MUST_BE_STABLE]. What may vary (wording): [MAY_VARY].

Provide:
1. A procedure: run each input 5 times with identical settings; record outputs in a table (Input | Run 1 ... Run 5).
2. What to compare: decisions or labels (count agreement), extracted values (exact match), format compliance (yes or no), length (min and max), tone (spot check), and key facts (match with the source).
3. A metric: agreement rate per input = number of runs matching the most common answer divided by 5; flag inputs below 80 percent.
4. Diagnosis of inconsistent items: ambiguous instructions, borderline cases, unspecified tie-break rules; suggestions to add (explicit rules, examples, a decision order).
5. Settings: if the tool allows, lower randomness for tasks that need repeatability; note that identical output is still not guaranteed.
6. A rule for tasks that must be consistent (use structured output, add validation code, or make the model show its criteria).
7. A results table template and a report format.

Give an example calculation with made-up results and a list of changes to try if the consistency is low.

Fill in before use

Optional: type your details and the prompt above updates. Nothing is saved or sent.

Replace each bracketed placeholder with your own details, then paste into Any AI or another assistant.

Input | Run 1 | Run 2 | Run 3 | Run 4 | Run 5 | Agreement
Ticket A | bug | bug | bug | bug | bug | 100%
Ticket B | billing | billing | how-to | billing | billing | 80%
Ticket C | unclear | bug | unclear | how-to | unclear | 60% (flag)
Agreement = most common answer count / 5. Diagnosis for C: ambiguous wording; add a rule "if no steps to reproduce are given, use unclear". Lower randomness settings if available; identical output is still not guaranteed.

• consistency