Meta Learning as Bayes Risk Minimization

TL;DR

This paper introduces a probabilistic framework for meta-learning, framing it as Bayesian risk minimization, and proposes a Gaussian approximation that improves posterior estimation in Neural Processes, validated on benchmarks.

Contribution

It formalizes meta-learning as Bayesian risk minimization and develops a Gaussian approximation for the posterior that enhances Neural Process models.

Findings

The Gaussian approximation converges to the maximum likelihood estimate at the same rate as the true posterior.

The proposed method outperforms existing models on benchmark datasets.

The framework justifies the Neural Process philosophy through Bayesian principles.

Abstract

Meta-Learning is a family of methods that use a set of interrelated tasks to learn a model that can quickly learn a new query task from a possibly small contextual dataset. In this study, we use a probabilistic framework to formalize what it means for two tasks to be related and reframe the meta-learning problem into the problem of Bayesian risk minimization (BRM). In our formulation, the BRM optimal solution is given by the predictive distribution computed from the posterior distribution of the task-specific latent variable conditioned on the contextual dataset, and this justifies the philosophy of Neural Process. However, the posterior distribution in Neural Process violates the way the posterior distribution changes with the contextual dataset. To address this problem, we present a novel Gaussian approximation for the posterior distribution that generalizes the posterior of the linear Gaussian model. Unlike that of the Neural Process, our approximation of the posterior distributions converges to the maximum likelihood estimate with the same rate as the true posterior distribution. We also demonstrate the competitiveness of our approach on benchmark datasets.