Meta-Learning with Latent Embedding Optimization

TL;DR

This paper introduces Latent Embedding Optimization (LEO), a meta-learning method that operates in a learned low-dimensional latent space to improve few-shot learning and fast adaptation, overcoming high-dimensional parameter challenges.

Contribution

The paper proposes a novel latent space approach for gradient-based meta-learning, enabling more effective adaptation in low-data regimes and high-dimensional models.

Findings

LEO achieves state-of-the-art results on miniImageNet and tieredImageNet.

LEO captures data uncertainty and improves adaptation efficiency.

Latent space optimization enhances meta-learning performance.

Abstract

Gradient-based meta-learning techniques are both widely applicable and proficient at solving challenging few-shot learning and fast adaptation problems. However, they have practical difficulties when operating on high-dimensional parameter spaces in extreme low-data regimes. We show that it is possible to bypass these limitations by learning a data-dependent latent generative representation of model parameters, and performing gradient-based meta-learning in this low-dimensional latent space. The resulting approach, latent embedding optimization (LEO), decouples the gradient-based adaptation procedure from the underlying high-dimensional space of model parameters. Our evaluation shows that LEO can achieve state-of-the-art performance on the competitive miniImageNet and tieredImageNet few-shot classification tasks. Further analysis indicates LEO is able to capture uncertainty in the data, and can perform adaptation more effectively by optimizing in latent space.