EnsLoss: Stochastic Calibrated Loss Ensembles for Preventing Overfitting in Classification

TL;DR

EnsLoss introduces a novel ensemble method that combines calibrated loss functions within the ERM framework, using stochastic gradient descent, to prevent overfitting and improve classification accuracy across diverse datasets.

Contribution

The paper proposes EnsLoss, a new ensemble approach that combines loss functions via calibrated loss-derivatives, ensuring statistical consistency and practical effectiveness in classification tasks.

Findings

EnsLoss outperforms fixed loss methods on multiple datasets.

Theoretical proof of statistical consistency for EnsLoss.

Effective in both tabular and image classification tasks.

Abstract

Empirical risk minimization (ERM) with a computationally feasible surrogate loss is a widely accepted approach for classification. Notably, the convexity and calibration (CC) properties of a loss function ensure consistency of ERM in maximizing accuracy, thereby offering a wide range of options for surrogate losses. In this article, we propose a novel ensemble method, namely EnsLoss, which extends the ensemble learning concept to combine loss functions within the ERM framework. A key feature of our method is the consideration on preserving the "legitimacy" of the combined losses, i.e., ensuring the CC properties. Specifically, we first transform the CC conditions of losses into loss-derivatives, thereby bypassing the need for explicit loss functions and directly generating calibrated loss-derivatives. Therefore, inspired by Dropout, EnsLoss enables loss ensembles through one training process with doubly stochastic gradient descent (i.e., random batch samples and random calibrated loss-derivatives). We theoretically establish the statistical consistency of our approach and provide insights into its benefits. The numerical effectiveness of EnsLoss compared to fixed loss methods is demonstrated through experiments on a broad range of 14 OpenML tabular datasets and 46 image datasets with various deep learning architectures. Python repository and source code are available on GitHub at https://github.com/statmlben/ensloss.