Large-Scale Meta-Learning with Continual Trajectory Shifting

TL;DR

This paper introduces a novel meta-learning approach that effectively handles large-scale, heterogeneous tasks by allowing more inner gradient steps and estimating parameter shifts, leading to improved convergence and performance.

Contribution

It proposes a method to increase meta-update frequency via task-specific parameter shift estimation, enhancing large-scale meta-learning effectiveness.

Findings

Outperforms previous first-order meta-learning methods in generalization.

Achieves better convergence and initialization quality.

Excels in multi-task learning and fine-tuning scenarios.

Abstract

Meta-learning of shared initialization parameters has shown to be highly effective in solving few-shot learning tasks. However, extending the framework to many-shot scenarios, which may further enhance its practicality, has been relatively overlooked due to the technical difficulties of meta-learning over long chains of inner-gradient steps. In this paper, we first show that allowing the meta-learners to take a larger number of inner gradient steps better captures the structure of heterogeneous and large-scale task distributions, thus results in obtaining better initialization points. Further, in order to increase the frequency of meta-updates even with the excessively long inner-optimization trajectories, we propose to estimate the required shift of the task-specific parameters with respect to the change of the initialization parameters. By doing so, we can arbitrarily increase the frequency of meta-updates and thus greatly improve the meta-level convergence as well as the quality of the learned initializations. We validate our method on a heterogeneous set of large-scale tasks and show that the algorithm largely outperforms the previous first-order meta-learning methods in terms of both generalization performance and convergence, as well as multi-task learning and fine-tuning baselines.