Binomial Gradient-Based Meta-Learning for Enhanced Meta-Gradient Estimation

TL;DR

This paper introduces BinomGBML, a novel meta-gradient estimation method using binomial expansion, improving accuracy and efficiency in gradient-based meta-learning, with theoretical error bounds and empirical performance gains.

Contribution

It proposes a binomial expansion approach for meta-gradient estimation in GBML, reducing approximation errors and enhancing scalability over existing truncated methods.

Findings

BinomGBML provides super-exponentially decaying error bounds.

Numerical tests show improved performance with manageable computational overhead.

Applied to MAML, it achieves better accuracy and efficiency.

Abstract

Meta-learning offers a principled framework leveraging \emph{task-invariant} priors from related tasks, with which \emph{task-specific} models can be fine-tuned on downstream tasks, even with limited data records. Gradient-based meta-learning (GBML) relies on gradient descent (GD) to adapt the prior to a new task. Albeit effective, these methods incur high computational overhead that scales linearly with the number of GD steps. To enhance efficiency and scalability, existing methods approximate the gradient of prior parameters (meta-gradient) via truncated backpropagation, yet suffer large approximation errors. Targeting accurate approximation, this work puts forth binomial GBML (BinomGBML), which relies on a truncated binomial expansion for meta-gradient estimation. This novel expansion endows more information in the meta-gradient estimation via efficient parallel computation. As a running paradigm applied to model-agnostic meta-learning (MAML), the resultant BinomMAML provably enjoys error bounds that not only improve upon existing approaches, but also decay super-exponentially under mild conditions. Numerical tests corroborate the theoretical analysis and showcase boosted performance with slightly increased computational overhead.