Learning under Distribution Mismatch and Model Misspecification

TL;DR

This paper investigates the impact of distribution mismatch and model misspecification on learning algorithms, connecting generalization error with rate-distortion theory to derive improved bounds and exploring auxiliary loss functions for better error control.

Contribution

It introduces a novel connection between generalization error and rate-distortion theory, providing improved bounds and methods to handle distribution mismatch and model misspecification.

Findings

Rate-distortion bounds improve upon previous bounds even without mismatch.

The connection enables new bounds on generalization error.

Auxiliary loss functions can be used to bound generalization error.

Abstract

We study learning algorithms when there is a mismatch between the distributions of the training and test datasets of a learning algorithm. The effect of this mismatch on the generalization error and model misspecification are quantified. Moreover, we provide a connection between the generalization error and the rate-distortion theory, which allows one to utilize bounds from the rate-distortion theory to derive new bounds on the generalization error and vice versa. In particular, the rate-distortion based bound strictly improves over the earlier bound by Xu and Raginsky even when there is no mismatch. We also discuss how "auxiliary loss functions" can be utilized to obtain upper bounds on the generalization error.