Filtering instances and rejecting predictions to obtain reliable models in healthcare

TL;DR

This paper presents a two-step data-centric method combining instance filtering and confidence-based rejection to improve the reliability of machine learning models in healthcare, ensuring safer and more trustworthy predictions.

Contribution

It introduces a novel approach that integrates Instance Hardness filtering with confidence-based rejection to enhance model reliability in healthcare applications.

Findings

Improved model reliability with high rejection rates.

Effective filtering of problematic instances during training.

Enhanced prediction confidence in real-world healthcare datasets.

Abstract

Machine Learning (ML) models are widely used in high-stakes domains such as healthcare, where the reliability of predictions is critical. However, these models often fail to account for uncertainty, providing predictions even with low confidence. This work proposes a novel two-step data-centric approach to enhance the performance of ML models by improving data quality and filtering low-confidence predictions. The first step involves leveraging Instance Hardness (IH) to filter problematic instances during training, thereby refining the dataset. The second step introduces a confidence-based rejection mechanism during inference, ensuring that only reliable predictions are retained. We evaluate our approach using three real-world healthcare datasets, demonstrating its effectiveness at improving model reliability while balancing predictive performance and rejection rate. Additionally, we use alternative criteria - influence values for filtering and uncertainty for rejection - as baselines to evaluate the efficiency of the proposed method. The results demonstrate that integrating IH filtering with confidence-based rejection effectively enhances model performance while preserving a large proportion of instances. This approach provides a practical method for deploying ML systems in safety-critical applications.