Frustratingly Easy Model Generalization by Dummy Risk Minimization

TL;DR

This paper introduces Dummy Risk Minimization (DuRM), a simple yet effective method to enhance the generalization of empirical risk minimization by enlarging output logits, validated through theoretical and extensive empirical analysis across diverse tasks.

Contribution

The paper proposes DuRM, a straightforward technique that improves ERM generalization by increasing output dimension, supported by theoretical insights and broad empirical validation.

Findings

DuRM increases gradient variance, aiding flat minima discovery.

DuRM consistently improves performance across various tasks and datasets.

DuRM is compatible with existing generalization methods.

Abstract

Empirical risk minimization (ERM) is a fundamental machine learning paradigm. However, its generalization ability is limited in various tasks. In this paper, we devise Dummy Risk Minimization (DuRM), a frustratingly easy and general technique to improve the generalization of ERM. DuRM is extremely simple to implement: just enlarging the dimension of the output logits and then optimizing using standard gradient descent. Moreover, we validate the efficacy of DuRM on both theoretical and empirical analysis. Theoretically, we show that DuRM derives greater variance of the gradient, which facilitates model generalization by observing better flat local minima. Empirically, we conduct evaluations of DuRM across different datasets, modalities, and network architectures on diverse tasks, including conventional classification, semantic segmentation, out-of-distribution generalization, adverserial training, and long-tailed recognition. Results demonstrate that DuRM could consistently improve the performance under all tasks with an almost free lunch manner. Furthermore, we show that DuRM is compatible with existing generalization techniques and we discuss possible limitations. We hope that DuRM could trigger new interest in the fundamental research on risk minimization.