Create intermediate Works with: Any AI

Evaluation Metrics Chooser

Few shot Has example

Pick metrics that match the real costs.

The Open buttons send the prompt text as currently shown below to that site only when you click. Fill in the placeholders first.

Choose evaluation metrics for my model. Task type (classification, regression, ranking): [TASK_TYPE]. Class balance or target distribution: [DISTRIBUTION]. Costs of different errors: [ERROR_COSTS]. Audience for the results: [AUDIENCE].

Produce:
1. A table: Metric | What it measures | Strength | Weakness | Suits my case? Include accuracy, precision, recall, F1, ROC AUC, PR AUC, log loss, calibration, MAE, RMSE, MAPE, R squared, as relevant to my task.
2. A recommended primary metric and two supporting metrics, with the reason tied to the cost of errors.
3. A worked example using a confusion matrix of invented numbers (for example 50 true positives, 10 false positives, 30 false negatives, 910 true negatives): compute accuracy, precision, recall, and F1 with arithmetic, and show why accuracy misleads on imbalanced data.
4. How to set a decision threshold using costs: choose the threshold that minimizes expected cost, with a table of three thresholds.
5. Subgroup evaluation: report metrics by relevant groups to detect uneven performance.
6. A baseline comparison.
7. The code to compute everything with scikit-learn.

Fill in before use

Optional: type your details and the prompt above updates. Nothing is saved or sent.

Replace each bracketed placeholder with your own details, then paste into Any AI or another assistant.

Task: fraud detection, 0.5% positives.
Accuracy is misleading: always predicting "not fraud" is 99.5% accurate.
Candidates: precision, recall, F1, precision-recall AUC.
Cost view: missed fraud costs 200; a false alert costs 5 of analyst time.
Choice: recall at a fixed precision of at least 20%, reported with PR curve; threshold chosen from costs.
Table
Metric | Good for | Weakness
ROC AUC | ranking | optimistic with rare positives
PR AUC | rare positives | harder to explain
Report: confidence intervals via cross-validation; slice by region.

• ml-metrics