Generalized Learning with Rejection for Classification and Regression Problems

TL;DR

This paper introduces a neural framework for learning with rejection applicable to both classification and regression, enabling models to abstain from low-confidence predictions, and demonstrates its effectiveness on synthetic, benchmark, and real-world data.

Contribution

The work presents a generalized neural meta-loss framework for learning with rejection that handles both classification and regression tasks, improving over existing methods.

Findings

Performs at par or better than existing methods on benchmark datasets.

Shows significant improvement in hurricane intensity prediction from satellite imagery.

Applicable to a wide range of machine learning tasks beyond classification.

Abstract

Learning with rejection (LWR) allows development of machine learning systems with the ability to discard low confidence decisions generated by a prediction model. That is, just like human experts, LWR allows machine models to abstain from generating a prediction when reliability of the prediction is expected to be low. Several frameworks for this learning with rejection have been proposed in the literature. However, most of them work for classification problems only and regression with rejection has not been studied in much detail. In this work, we present a neural framework for LWR based on a generalized meta-loss function that involves simultaneous training of two neural network models: a predictor model for generating predictions and a rejecter model for deciding whether the prediction should be accepted or rejected. The proposed framework can be used for classification as well as regression and other related machine learning tasks. We have demonstrated the applicability and effectiveness of the method on synthetically generated data as well as benchmark datasets from UCI machine learning repository for both classification and regression problems. Despite being simpler in implementation, the proposed scheme for learning with rejection has shown to perform at par or better than previously proposed methods. Furthermore, we have applied the method to the problem of hurricane intensity prediction from satellite imagery. Significant improvement in performance as compared to conventional supervised methods shows the effectiveness of the proposed scheme in real-world regression problems.