Learning to Abstain From Uninformative Data

TL;DR

This paper introduces a novel method for learning in noisy environments with high uninformative data, ensuring near-optimal decisions by distinguishing informative samples during training and testing.

Contribution

It proposes a new loss function and an iterative algorithm to jointly optimize a predictor and a selector for handling uninformative data effectively.

Findings

The method guarantees near-optimal decision-making under high noise conditions.

Empirical results show improved performance over existing approaches.

The approach is applicable in domains like finance and healthcare.

Abstract

Learning and decision-making in domains with naturally high noise-to-signal ratio, such as Finance or Healthcare, is often challenging, while the stakes are very high. In this paper, we study the problem of learning and acting under a general noisy generative process. In this problem, the data distribution has a significant proportion of uninformative samples with high noise in the label, while part of the data contains useful information represented by low label noise. This dichotomy is present during both training and inference, which requires the proper handling of uninformative data during both training and testing. We propose a novel approach to learning under these conditions via a loss inspired by the selective learning theory. By minimizing this loss, the model is guaranteed to make a near-optimal decision by distinguishing informative data from uninformative data and making predictions. We build upon the strength of our theoretical guarantees by describing an iterative algorithm, which jointly optimizes both a predictor and a selector, and evaluates its empirical performance in a variety of settings.