RELF: Robust Regression Extended with Ensemble Loss Function

TL;DR

This paper introduces RELF, a robust ensemble loss function for regression that enhances performance and robustness in noisy environments through a novel half-quadratic learning algorithm.

Contribution

It proposes a new ensemble loss function for regression, along with a half-quadratic algorithm to optimize parameters and weights, demonstrating improved robustness and accuracy.

Findings

Significantly improves regression performance over state-of-the-art methods.

Proves robustness of the ensemble loss in noisy environments.

Shows Bayes consistency for a class of loss functions.

Abstract

Ensemble techniques are powerful approaches that combine several weak learners to build a stronger one. As a meta-learning framework, ensemble techniques can easily be applied to many machine learning methods. Inspired by ensemble techniques, in this paper we propose an ensemble loss functions applied to a simple regressor. We then propose a half-quadratic learning algorithm in order to find the parameter of the regressor and the optimal weights associated with each loss function. Moreover, we show that our proposed loss function is robust in noisy environments. For a particular class of loss functions, we show that our proposed ensemble loss function is Bayes consistent and robust. Experimental evaluations on several datasets demonstrate that our proposed ensemble loss function significantly improves the performance of a simple regressor in comparison with state-of-the-art methods.