Generalization Bounds with Minimal Dependency on Hypothesis Class via Distributionally Robust Optimization

TL;DR

This paper introduces a novel approach to derive generalization bounds in machine learning using distributionally robust optimization (DRO), which depend minimally on the hypothesis class and are based on ambiguity set geometry.

Contribution

The paper presents the first generalization bounds for DRO that depend only on the true loss function, reducing reliance on hypothesis class complexity.

Findings

DRO bounds depend on ambiguity set geometry and true loss function.

Using statistical distances like Wasserstein yields minimal hypothesis class dependence.

First known bounds of this type in the literature.

Abstract

Established approaches to obtain generalization bounds in data-driven optimization and machine learning mostly build on solutions from empirical risk minimization (ERM), which depend crucially on the functional complexity of the hypothesis class. In this paper, we present an alternate route to obtain these bounds on the solution from distributionally robust optimization (DRO), a recent data-driven optimization framework based on worst-case analysis and the notion of ambiguity set to capture statistical uncertainty. In contrast to the hypothesis class complexity in ERM, our DRO bounds depend on the ambiguity set geometry and its compatibility with the true loss function. Notably, when using statistical distances such as maximum mean discrepancy, Wasserstein distance, or $\phi$-divergence in the DRO, our analysis implies generalization bounds whose dependence on the hypothesis class appears the minimal possible: The bound depends solely on the true loss function, independent of any other candidates in the hypothesis class. To our best knowledge, it is the first generalization bound of this type in the literature, and we hope our findings can open the door for a better understanding of DRO, especially its benefits on loss minimization and other machine learning applications.