On the Generalization Error of Meta Learning for the Gibbs Algorithm

TL;DR

This paper provides an exact information-theoretic analysis of the generalization error in meta learning algorithms based on the Gibbs framework, offering new bounds and characterizations.

Contribution

It introduces a novel symmetrized KL information-based analysis for the meta Gibbs algorithm and derives distribution-free generalization bounds.

Findings

Exact characterization of meta generalization error using symmetrized KL information.

Derived bounds applicable to various meta learning scenarios.

Extended analysis to super-task Gibbs algorithms with conditional information measures.

Abstract

We analyze the generalization ability of joint-training meta learning algorithms via the Gibbs algorithm. Our exact characterization of the expected meta generalization error for the meta Gibbs algorithm is based on symmetrized KL information, which measures the dependence between all meta-training datasets and the output parameters, including task-specific and meta parameters. Additionally, we derive an exact characterization of the meta generalization error for the super-task Gibbs algorithm, in terms of conditional symmetrized KL information within the super-sample and super-task framework introduced in Steinke and Zakynthinou (2020) and Hellstrom and Durisi (2022) respectively. Our results also enable us to provide novel distribution-free generalization error upper bounds for these Gibbs algorithms applicable to meta learning.