# Generalization error for decision problems

## TL;DR

This paper reviews different definitions of generalization error in classification and decision problems, highlighting their differences, and proposes data-adaptive methods for valid confidence estimation, emphasizing the importance of proper asymptotic techniques.

## Contribution

It clarifies the non-equivalence of various generalization error definitions and introduces data-adaptive procedures for accurate confidence set construction.

## Key findings

- Different definitions of generalization error can be asymptotically non-equivalent.
- Standard asymptotic methods may fail for non-smooth functionals like generalization error.
- Proposed data-adaptive procedures yield asymptotically valid confidence sets.

## Abstract

In this entry we review the generalization error for classification and single-stage decision problems. We distinguish three alternative definitions of the generalization error which have, at times, been conflated in the statistics literature and show that these definitions need not be equivalent even asymptotically. Because the generalization error is a non-smooth functional of the underlying generative model, standard asymptotic approximations, e.g., the bootstrap or normal approximations, cannot guarantee correct frequentist operating characteristics without modification. We provide simple data-adaptive procedures that can be used to construct asymptotically valid confidence sets for the generalization error. We conclude the entry with a discussion of extensions and related problems.