Diametrical Risk Minimization: Theory and Computations

TL;DR

This paper introduces Diametrical Risk Minimization (DRM), a novel approach that improves generalization in machine learning by considering worst-case risks in parameter neighborhoods, outperforming traditional ERM especially in complex landscapes.

Contribution

The paper develops DRM, providing theoretical bounds independent of Lipschitz constants and practical algorithms, addressing ERM's limitations in sharp risk landscapes.

Findings

DRM achieves low generalization error in neural network classification.

DRM's bounds are independent of Lipschitz moduli for convex and nonconvex problems.

Numerical results show DRM outperforms ERM in corrupted label scenarios.

Abstract

The theoretical and empirical performance of Empirical Risk Minimization (ERM) often suffers when loss functions are poorly behaved with large Lipschitz moduli and spurious sharp minimizers. We propose and analyze a counterpart to ERM called Diametrical Risk Minimization (DRM), which accounts for worst-case empirical risks within neighborhoods in parameter space. DRM has generalization bounds that are independent of Lipschitz moduli for convex as well as nonconvex problems and it can be implemented using a practical algorithm based on stochastic gradient descent. Numerical results illustrate the ability of DRM to find quality solutions with low generalization error in sharp empirical risk landscapes from benchmark neural network classification problems with corrupted labels.