Designing a Robust, Bounded, and Smooth Loss Function for Improved Supervised Learning

TL;DR

This paper introduces RoBoS-NN, a new loss function designed to enhance supervised learning by being robust, bounded, and smooth, thereby improving performance especially with outlier-prone, high-dimensional data.

Contribution

The paper proposes a novel loss function, RoBoS-NN, with theoretical analysis and demonstrates its effectiveness in neural networks for time series forecasting, especially in challenging scenarios.

Findings

RoBoS-NN outperforms benchmark models in accuracy.

The loss function is robust against outliers.

Theoretical analysis confirms its generalization ability.

Abstract

The loss function is crucial to machine learning, especially in supervised learning frameworks. It is a fundamental component that controls the behavior and general efficacy of learning algorithms. However, despite their widespread use, traditional loss functions have significant drawbacks when dealing with high-dimensional and outlier-sensitive datasets, which frequently results in reduced performance and slower convergence during training. In this work, we develop a robust, bounded, and smooth (RoBoS-NN) loss function to resolve the aforementioned hindrances. The generalization ability of the loss function has also been theoretically analyzed to rigorously justify its robustness. Moreover, we implement RoboS-NN loss in the framework of a neural network (NN) to forecast time series and present a new robust algorithm named $\mathcal{L}_{\text{RoBoS}}$-NN. To assess the potential of $\mathcal{L}_{\text{RoBoS}}$-NN, we conduct experiments on multiple real-world datasets. In addition, we infuse outliers into data sets to evaluate the performance of $\mathcal{L}_{\text{RoBoS}}$-NN in more challenging scenarios. Numerical results show that $\mathcal{L}_{\text{RoBoS}}$-NN outperforms the other benchmark models in terms of accuracy measures.