Wormhole MAML: Meta-Learning in Glued Parameter Space

TL;DR

Wormhole MAML introduces a multiplicative parameter in meta-learning to create a shortcut in parameter space, enhancing expressivity and training stability across diverse tasks.

Contribution

It proposes a novel variation of MAML with an extra parameter, improving gradient conflict issues and training dynamics, supported by theoretical and empirical evidence.

Findings

Alleviates conflicting gradients in meta-learning

Enhances training stability and expressivity

Effective across diverse problem types

Abstract

In this paper, we introduce a novel variation of model-agnostic meta-learning, where an extra multiplicative parameter is introduced in the inner-loop adaptation. Our variation creates a shortcut in the parameter space for the inner-loop adaptation and increases model expressivity in a highly controllable manner. We show both theoretically and numerically that our variation alleviates the problem of conflicting gradients and improves training dynamics. We conduct experiments on 3 distinctive problems, including a toy classification problem for threshold comparison, a regression problem for wavelet transform, and a classification problem on MNIST. We also discuss ways to generalize our method to a broader class of problems.