Outlier-robust neural network training: variation regularization meets trimmed loss to prevent functional breakdown

TL;DR

This paper proposes a novel training method for neural networks that combines a transformed trimmed loss with higher-order variation regularization to enhance robustness against outliers, backed by theoretical guarantees.

Contribution

It introduces a new outlier-robust training framework for neural networks that controls model capacity and maintains high robustness, extending classical robust statistics to nonlinear models.

Findings

The method retains a high functional breakdown point.

The approach effectively suppresses overfitting to outliers.

The stochastic optimization algorithm converges theoretically.

Abstract

In this study, we tackle the challenge of outlier-robust predictive modeling using highly expressive neural networks. Our approach integrates two key components: (1) a transformed trimmed loss (TTL), a computationally efficient variant of the classical trimmed loss, and (2) higher-order variation regularization (HOVR), which imposes smoothness constraints on the prediction function. While traditional robust statistics typically assume low-complexity models such as linear and kernel models, applying TTL alone to modern neural networks may fail to ensure robustness, as their high expressive power allows them to fit both inliers and outliers, even when a robust loss is used. To address this, we revisit the traditional notion of breakdown point and adapt it to the nonlinear function setting, introducing a regularization scheme via HOVR that controls the model's capacity and suppresses overfitting to outliers. We theoretically establish that our training procedure retains a high functional breakdown point, thereby ensuring robustness to outlier contamination. We develop a stochastic optimization algorithm tailored to this framework and provide a theoretical guarantee of its convergence.