Z-Error Loss for Training Neural Networks

TL;DR

The paper introduces Z-Error Loss, a novel training method that reduces the impact of outliers in neural networks by masking out-of-distribution samples during batch training, improving robustness and data diagnostics.

Contribution

It presents a statistically principled loss function that automatically detects and excludes outliers during training, enhancing model robustness and data quality assessment.

Findings

Improves model robustness against outliers

Enables automatic outlier detection during training

Provides diagnostics for data curation

Abstract

Outliers introduce significant training challenges in neural networks by propagating erroneous gradients, which can degrade model performance and generalization. We propose the Z-Error Loss, a statistically principled approach that minimizes outlier influence during training by masking the contribution of data points identified as out-of-distribution within each batch. This method leverages batch-level statistics to automatically detect and exclude anomalous samples, allowing the model to focus its learning on the true underlying data structure. Our approach is robust, adaptive to data quality, and provides valuable diagnostics for data curation and cleaning.