Learning to be Reproducible: Custom Loss Design for Robust Neural Networks

TL;DR

This paper introduces a custom loss function that improves the robustness and reproducibility of neural network training, ensuring more consistent performance across different runs without compromising accuracy.

Contribution

The paper proposes a novel custom loss function (CLF) that enhances training stability and reproducibility in neural networks, addressing variability caused by stochastic training factors.

Findings

CLF reduces performance variability across training runs.

Extensive experiments show improved robustness without accuracy loss.

Applicable to diverse architectures and tasks.

Abstract

To enhance the reproducibility and reliability of deep learning models, we address a critical gap in current training methodologies: the lack of mechanisms that ensure consistent and robust performance across runs. Our empirical analysis reveals that even under controlled initialization and training conditions, the accuracy of the model can exhibit significant variability. To address this issue, we propose a Custom Loss Function (CLF) that reduces the sensitivity of training outcomes to stochastic factors such as weight initialization and data shuffling. By fine-tuning its parameters, CLF explicitly balances predictive accuracy with training stability, leading to more consistent and reliable model performance. Extensive experiments across diverse architectures for both image classification and time series forecasting demonstrate that our approach significantly improves training robustness without sacrificing predictive performance. These results establish CLF as an effective and efficient strategy for developing more stable, reliable and trustworthy neural networks.