When MAML Can Adapt Fast and How to Assist When It Cannot

TL;DR

This paper investigates how MAML adapts to new tasks, revealing that deep architectures enhance adaptation and proposing new meta-optimization methods that outperform MAML in various architectures.

Contribution

It provides empirical and theoretical insights into MAML's adaptation mechanisms and introduces novel meta-optimization approaches that improve performance.

Findings

Deep architectures improve MAML's adaptation even for shallow tasks.

Upper layers in deep networks facilitate fast adaptation through meta-learned gradient updates.

Proposed meta-optimization methods outperform MAML in different architectures.

Abstract

Model-Agnostic Meta-Learning (MAML) and its variants have achieved success in meta-learning tasks on many datasets and settings. On the other hand, we have just started to understand and analyze how they are able to adapt fast to new tasks. For example, one popular hypothesis is that the algorithms learn good representations for transfer, as in multi-task learning. In this work, we contribute by providing a series of empirical and theoretical studies, and discover several interesting yet previously unknown properties of the algorithm. We find MAML adapts better with a deep architecture even if the tasks need only a shallow one (and thus, no representation learning is needed). While echoing previous findings by others that the bottom layers in deep architectures enable representation learning, we also find that upper layers enable fast adaptation by being meta-learned to perform adaptive gradient update when generalizing to new tasks. Motivated by these findings, we study several meta-optimization approaches and propose a new one for learning to optimize adaptively. Those approaches attain stronger performance in meta-learning both shallower and deeper architectures than MAML.