Meta Learning in the Continuous Time Limit

TL;DR

This paper derives an ODE framework for understanding MAML training dynamics, revealing convergence properties and leading to a new, more efficient training algorithm validated by empirical results.

Contribution

It introduces a continuous-time ODE perspective for MAML, proving convergence and proposing a novel, computationally efficient training method.

Findings

MAML dynamics can be described by an underlying ODE.

The MAML ODE converges linearly to stationary points.

BI-MAML reduces computational costs significantly.

Abstract

In this paper, we establish the ordinary differential equation (ODE) that underlies the training dynamics of Model-Agnostic Meta-Learning (MAML). Our continuous-time limit view of the process eliminates the influence of the manually chosen step size of gradient descent and includes the existing gradient descent training algorithm as a special case that results from a specific discretization. We show that the MAML ODE enjoys a linear convergence rate to an approximate stationary point of the MAML loss function for strongly convex task losses, even when the corresponding MAML loss is non-convex. Moreover, through the analysis of the MAML ODE, we propose a new BI-MAML training algorithm that significantly reduces the computational burden associated with existing MAML training methods. To complement our theoretical findings, we perform empirical experiments to showcase the superiority of our proposed methods with respect to the existing work.