Information Theoretic Meta Learning with Gaussian Processes

TL;DR

This paper introduces an information theoretic framework for meta learning that leverages Gaussian processes for non-parametric task encoding, unifying and extending existing algorithms with competitive results.

Contribution

It presents a novel information theoretic approach to meta learning, incorporating Gaussian processes for non-parametric task representations and deriving new algorithms.

Findings

Achieved competitive accuracy on few-shot classification tasks.

Unified existing gradient-based meta learning algorithms within a new information theoretic framework.

Developed a memory-based Gaussian process algorithm for meta learning.

Abstract

We formulate meta learning using information theoretic concepts; namely, mutual information and the information bottleneck. The idea is to learn a stochastic representation or encoding of the task description, given by a training set, that is highly informative about predicting the validation set. By making use of variational approximations to the mutual information, we derive a general and tractable framework for meta learning. This framework unifies existing gradient-based algorithms and also allows us to derive new algorithms. In particular, we develop a memory-based algorithm that uses Gaussian processes to obtain non-parametric encoding representations. We demonstrate our method on a few-shot regression problem and on four few-shot classification problems, obtaining competitive accuracy when compared to existing baselines.