Generalization Bounds for Vicinal Risk Minimization Principle

TL;DR

This paper provides a theoretical analysis of the generalization bounds for vicinal risk minimization (VRM), showing how the choice of vicinal functions influences learning performance and offering insights into existing VRM models.

Contribution

It establishes the first comprehensive theoretical bounds for VRM, linking function class complexity, vicinal functions, and generalization performance.

Findings

Function class complexity convolved with vicinal functions can be controlled by original class complexity.

Generalization bounds depend on the choice and quality of vicinal functions.

Theoretical explanations for existing VRM models like mixup and Gaussian-based models.

Abstract

The vicinal risk minimization (VRM) principle, first proposed by \citet{vapnik1999nature}, is an empirical risk minimization (ERM) variant that replaces Dirac masses with vicinal functions. Although there is strong numerical evidence showing that VRM outperforms ERM if appropriate vicinal functions are chosen, a comprehensive theoretical understanding of VRM is still lacking. In this paper, we study the generalization bounds for VRM. Our results support Vapnik's original arguments and additionally provide deeper insights into VRM. First, we prove that the complexity of function classes convolving with vicinal functions can be controlled by that of the original function classes under the assumption that the function class is composed of Lipschitz-continuous functions. Then, the resulting generalization bounds for VRM suggest that the generalization performance of VRM is also effected by the choice of vicinity function and the quality of function classes. These findings can be used to examine whether the choice of vicinal function is appropriate for the VRM-based learning setting. Finally, we provide a theoretical explanation for existing VRM models, e.g., uniform distribution-based models, Gaussian distribution-based models, and mixup models.