Novel Techniques to Assess Predictive Systems and Reduce Their Alarm Burden
Jonathan A. Handler, Craig F. Feied, Michael T. Gillam

TL;DR
This paper introduces utility-based performance metrics for predictive systems, especially in clinical settings, addressing limitations of traditional metrics by accounting for workflow context and temporal relationships.
Contribution
It presents a novel utility metric framework (u-metrics) that better reflects real-world predictor utility and introduces a formal snoozing approach to improve alarm systems.
Findings
u-metrics outperform traditional metrics in clinical prediction scenarios
Snoozing reduces false positives while maintaining event detection
Utility metrics accurately predict performance improvements from snoozing
Abstract
Machine prediction algorithms (e.g., binary classifiers) often are adopted on the basis of claimed performance using classic metrics such as sensitivity and predictive value. However, classifier performance depends heavily upon the context (workflow) in which the classifier operates. Classic metrics do not reflect the realized utility of a predictor unless certain implicit assumptions are met, and these assumptions cannot be met in many common clinical scenarios. This often results in suboptimal implementations and in disappointment when expected outcomes are not achieved. One common failure mode for classic metrics arises when multiple predictions can be made for the same event, particularly when redundant true positive predictions produce little additional value. This describes many clinical alerting systems. We explain why classic metrics cannot correctly represent predictor…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsMachine Learning in Healthcare · Healthcare Technology and Patient Monitoring · Data Stream Mining Techniques
