Probability Calibration Check
See whether predicted probabilities can be trusted.
The Open buttons send the prompt text as currently shown below to that site only when you click. Fill in the placeholders first.
Check the calibration of my classifier. Model, predicted probabilities and true labels summary: [PREDICTIONS_SUMMARY]. How probabilities are used in decisions: [DECISION_USE]. Explain calibration in plain terms. Provide code to produce a reliability curve and compute a calibration error and the Brier score, and interpret them. Describe fixes such as Platt scaling or isotonic regression, and how to apply them without leakage using a separate calibration set. Discuss when calibration matters (thresholds, costs, risk scores) and when ranking is enough. Include a checklist for monitoring calibration over time. Add a small invented example of predicted probabilities and outcomes in ten bins with the calculation of the reliability points, so that I can confirm my code. Include a note on sample size per bin and the risk of reading too much into noisy curves, and a monitoring routine for calibration drift. Also add a one-paragraph summary I could send to a non-technical stakeholder explaining what well calibrated means.
Fill in before use
Optional: type your details and the prompt above updates. Nothing is saved or sent.
Replace each bracketed placeholder with your own details, then paste into Any AI or another assistant.