Global quantitative robustness of regression feed-forward neural networks

TL;DR

This paper investigates the robustness of regression feed-forward neural networks by adapting classical robust statistics concepts, specifically the breakdown point, and demonstrates the benefits of using robust loss functions through extensive simulations.

Contribution

It introduces the adaptation of the regression breakdown point to neural networks and compares robust and non-robust networks across various configurations.

Findings

Robust neural networks show improved out-of-sample loss.

Using robust loss functions enhances network stability.

Breakdown points vary with network configuration and contamination.

Abstract

Neural networks are an indispensable model class for many complex learning tasks. Despite the popularity and importance of neural networks and many different established techniques from literature for stabilization and robustification of the training, the classical concepts from robust statistics have rarely been considered so far in the context of neural networks. Therefore, we adapt the notion of the regression breakdown point to regression neural networks and compute the breakdown point for different feed-forward network configurations and contamination settings. In an extensive simulation study, we compare the performance, measured by the out-of-sample loss, by a proxy of the breakdown rate and by the training steps, of non-robust and robust regression feed-forward neural networks in a plethora of different configurations. The results indeed motivate to use robust loss functions for neural network training.