Conditional Mutual Information-Based Generalization Bound for Meta Learning

TL;DR

This paper develops an information-theoretic generalization bound for meta-learning using conditional mutual information, providing a new way to quantify how well a meta-learner can generalize from limited task data.

Contribution

It extends the CMI framework to meta-learning, deriving an explicit bound based on a meta-supersample and demonstrating its advantages over previous bounds.

Findings

The bound explicitly involves two CMI terms measuring information about data selection.

Numerical example shows the bound's effectiveness compared to prior bounds.

The approach offers a new perspective on meta-learning generalization performance.

Abstract

Meta-learning optimizes an inductive bias---typically in the form of the hyperparameters of a base-learning algorithm---by observing data from a finite number of related tasks. This paper presents an information-theoretic bound on the generalization performance of any given meta-learner, which builds on the conditional mutual information (CMI) framework of Steinke and Zakynthinou (2020). In the proposed extension to meta-learning, the CMI bound involves a training \textit{meta-supersample} obtained by first sampling $2N$ independent tasks from the task environment, and then drawing $2M$ independent training samples for each sampled task. The meta-training data fed to the meta-learner is modelled as being obtained by randomly selecting $N$ tasks from the available $2N$ tasks and $M$ training samples per task from the available $2M$ training samples per task. The resulting bound is explicit in two CMI terms, which measure the information that the meta-learner output and the base-learner output provide about which training data are selected, given the entire meta-supersample. Finally, we present a numerical example that illustrates the merits of the proposed bound in comparison to prior information-theoretic bounds for meta-learning.