TRIAGE: Characterizing and auditing training data for improved regression

TL;DR

TRIAGE introduces a model-agnostic framework for characterizing and auditing training data in regression tasks, enhancing data quality assessment and improving model performance through data filtering and selection.

Contribution

The paper presents TRIAGE, a novel data characterization method for regression that uses conformal predictive distributions to analyze training data and improve regression models.

Findings

TRIAGE effectively characterizes training samples as under-, over-, or well-estimated.

Using TRIAGE improves regression performance through data filtering.

TRIAGE enables new dataset selection and feature acquisition strategies.

Abstract

Data quality is crucial for robust machine learning algorithms, with the recent interest in data-centric AI emphasizing the importance of training data characterization. However, current data characterization methods are largely focused on classification settings, with regression settings largely understudied. To address this, we introduce TRIAGE, a novel data characterization framework tailored to regression tasks and compatible with a broad class of regressors. TRIAGE utilizes conformal predictive distributions to provide a model-agnostic scoring method, the TRIAGE score. We operationalize the score to analyze individual samples' training dynamics and characterize samples as under-, over-, or well-estimated by the model. We show that TRIAGE's characterization is consistent and highlight its utility to improve performance via data sculpting/filtering, in multiple regression settings. Additionally, beyond sample level, we show TRIAGE enables new approaches to dataset selection and feature acquisition. Overall, TRIAGE highlights the value unlocked by data characterization in real-world regression applications