Provably robust learning of regression neural networks using $\beta$-divergences

TL;DR

This paper introduces rRNet, a robust regression neural network framework based on $eta$-divergence, providing theoretical robustness guarantees and demonstrating practical advantages over existing methods.

Contribution

It develops a broad, theoretically grounded robust learning method for regression NNs using $eta$-divergence, with convergence and robustness guarantees.

Findings

rRNet achieves bounded influence functions for suitable $eta$

It attains a 50 ext% asymptotic breakdown point

Simulation and real-data experiments show improved robustness

Abstract

Regression neural networks (NNs) are most commonly trained by minimizing the mean squared prediction error, which is highly sensitive to outliers and data contamination. Existing robust training methods for regression NNs are often limited in scope and rely primarily on empirical validation, with only a few offering partial theoretical guarantees. In this paper, we propose a new robust learning framework for regression NNs based on the $\beta$-divergence (also known as the density power divergence) which we call `rRNet'. It applies to a broad class of regression NNs, including models with non-smooth activation functions and error densities, and recovers the classical maximum likelihood learning as a special case. The rRNet is implemented via an alternating optimization scheme, for which we establish convergence guarantees to stationary points under mild, verifiable conditions. The (local) robustness of rRNet is theoretically characterized through the influence functions of both the parameter estimates and the resulting rRNet predictor, which are shown to be bounded for suitable choices of the tuning parameter $\beta$, depending on the error density. We further prove that rRNet attains the optimal 50\% asymptotic breakdown point at the assumed model for all $\beta\in(0, 1]$, providing a strong global robustness guarantee that is largely absent for existing NN learning methods. Our theoretical results are complemented by simulation experiments and real-data analyses, illustrating practical advantages of rRNet over existing approaches in both function approximation problems and prediction tasks with noisy observations.