Rubric for Chatbot Reply Quality
Score assistant replies on accuracy, helpfulness and safety.
The Open buttons send the prompt text as currently shown below to that site only when you click. Fill in the placeholders first.
Create a rubric for scoring replies from a chatbot that does [CHATBOT_JOB]. Its users: [USERS]. Rules it must follow: [RULES]. Typical failures I have seen: [FAILURES]. Define criteria: factual accuracy against the knowledge provided, answering the actual question, completeness, tone, safe handling of sensitive requests, and honesty about uncertainty. Score each 0 to 3 with written anchors and give two sample replies per score for one criterion, using invented content. Add a rule for scoring replies that refuse: a good refusal is polite, explains briefly and points to an alternative. Include a field for the grader's quoted evidence and a calculation for the average with the arithmetic shown. Recommend how many replies per week a person should score and how to sample them. Include a short list of safety checks that apply to every reply regardless of score: no request for passwords or full card numbers, no medical, legal or financial advice beyond the approved scope, and no claim of being a human. A reply that breaks one of these fails automatically, even if it scores highly elsewhere. Describe how to record the failure and who should be told, and propose a weekly report listing the top three failure patterns with examples quoted from the replies.
Fill in before use
Optional: type your details and the prompt above updates. Nothing is saved or sent.
Replace each bracketed placeholder with your own details, then paste into Any AI or another assistant.