DeepMoM: Robust Deep Learning With Median-of-Means

TL;DR

This paper introduces DeepMoM, a robust deep learning method based on median-of-means, designed to handle common data corruptions and measurement errors more effectively than traditional approaches.

Contribution

The paper proposes a novel median-of-means based approach for deep learning that improves robustness against random data corruptions, inspired by recent statistical insights.

Findings

Performs well in practice on corrupted data

Offers a promising alternative to standard training methods

Demonstrates robustness to measurement errors and unreliable sources

Abstract

Data used in deep learning is notoriously problematic. For example, data are usually combined from diverse sources, rarely cleaned and vetted thoroughly, and sometimes corrupted on purpose. Intentional corruption that targets the weak spots of algorithms has been studied extensively under the label of "adversarial attacks." In contrast, the arguably much more common case of corruption that reflects the limited quality of data has been studied much less. Such "random" corruptions are due to measurement errors, unreliable sources, convenience sampling, and so forth. These kinds of corruption are common in deep learning, because data are rarely collected according to strict protocols -- in strong contrast to the formalized data collection in some parts of classical statistics. This paper concerns such corruption. We introduce an approach motivated by very recent insights into median-of-means and Le Cam's principle, we show that the approach can be readily implemented, and we demonstrate that it performs very well in practice. In conclusion, we believe that our approach is a very promising alternative to standard parameter training based on least-squares and cross-entropy loss.