Memory-Reduced Meta-Learning with Guaranteed Convergence

TL;DR

This paper introduces a memory-efficient meta-learning algorithm that avoids using historical parameters, guarantees convergence, and matches the computational complexity of existing methods, with confirmed effectiveness on benchmarks.

Contribution

The paper proposes a novel meta-learning algorithm that reduces memory usage and provides convergence guarantees without relying on historical gradients or parameters.

Findings

Reduces memory costs compared to existing approaches.

Proves sublinear convergence and error decay.

Experimental results confirm effectiveness on benchmarks.

Abstract

The optimization-based meta-learning approach is gaining increased traction because of its unique ability to quickly adapt to a new task using only small amounts of data. However, existing optimization-based meta-learning approaches, such as MAML, ANIL and their variants, generally employ backpropagation for upper-level gradient estimation, which requires using historical lower-level parameters/gradients and thus increases computational and memory overhead in each iteration. In this paper, we propose a meta-learning algorithm that can avoid using historical parameters/gradients and significantly reduce memory costs in each iteration compared to existing optimization-based meta-learning approaches. In addition to memory reduction, we prove that our proposed algorithm converges sublinearly with the iteration number of upper-level optimization, and the convergence error decays sublinearly with the batch size of sampled tasks. In the specific case in terms of deterministic meta-learning, we also prove that our proposed algorithm converges to an exact solution. Moreover, we quantify that the computational complexity of the algorithm is on the order of $\mathcal{O}(\epsilon^{-1})$, which matches existing convergence results on meta-learning even without using any historical parameters/gradients. Experimental results on meta-learning benchmarks confirm the efficacy of our proposed algorithm.