Follow the idea.
A concept, in three framesInspect the scores.
Scores are supplied, not calibrated probabilities.
Move the cutoff.
The decision changes at an explicit boundary.
Count both errors.
False positives and false negatives differ.
A WORKING DEFINITION
Classification thresholds
A classification threshold maps a score to a decision. Here a message is flagged when its score is greater than or equal to the cutoff. Comparing that decision with a supplied label produces one of four outcomes.
FOLLOW ONE EXAMPLE
A cutoff is a choice with consequences.
At cutoff60, seven spam messages and two legitimate messages are flagged; one spam message is missed and ten legitimate messages pass. At80, the counts are three, one, five and eleven. Precision is TP/(TP+FP); recall is TP/(TP+FN). If nothing is flagged, precision has no denominator and remains undefined.
Work through the investigation →WHERE THE CLAIM STOPS
Keep the assumptions attached.
These scores are not calibrated probabilities, and twenty supplied messages do not estimate real-world performance. Assigned costs are fictional points, not objective harm.
Inspect this model’s assumptions and definitions
- All labels and scores are supplied; these are not empirical classifier performance estimates.
- Scores are not calibrated probabilities. Error costs are fictional points chosen by the reader.
- Changing costs does not move the threshold or retrain a model. Precision is undefined if no messages are flagged.
Definitions used in this instrument
- dataset
- twenty-supplied-messages-v1
- cutoff
- flag-score-greater-than-or-equal-to-threshold
- score
- integer-0-100-not-calibrated-probability
- cost
- FP-times-FP-cost-plus-FN-times-FN-cost
SOURCES & SCOPE
Read beyond the model.
Godil et al. (2014), Performance Metrics for Evaluating Object and Human Detection and Tracking Systems ↗
NISTIR7972 defines detection counts, precision and recall. Its application is detection and tracking; this page uses only the general metric definitions with an original fictional message set.
All example data, figures and exact calculations on this page are original constructed teaching material. Source publications provide conceptual background, not the example dataset.
Open the instrument →