Meta-Learning Neural Procedural Biases
Christian Raymond, Qi Chen, Bing Xue, Mengjie Zhang
TL;DR
This paper introduces Neural Procedural Bias Meta-Learning (NPBML), a framework that meta-learns task-specific procedural biases to improve few-shot learning performance across various benchmarks.
Contribution
It proposes a novel method to meta-learn task-adaptive procedural biases, enhancing the inductive biases for better few-shot learning outcomes.
Findings
Meta-learned procedural biases improve few-shot learning performance.
The approach outperforms existing methods on multiple benchmarks.
Procedural biases adapt to individual tasks for optimized learning.
Abstract
The goal of few-shot learning is to generalize and achieve high performance on new unseen learning tasks, where each task has only a limited number of examples available. Gradient-based meta-learning attempts to address this challenging task by learning how to learn new tasks by embedding inductive biases informed by prior learning experiences into the components of the learning algorithm. In this work, we build upon prior research and propose Neural Procedural Bias Meta-Learning (NPBML), a novel framework designed to meta-learn task-adaptive procedural biases. Our approach aims to consolidate recent advancements in meta-learned initializations, optimizers, and loss functions by learning them simultaneously and making them adapt to each individual task to maximize the strength of the learned inductive biases. This imbues each learning task with a unique set of procedural biases which is…
Peer Reviews
Decision·Submitted to ICLR 2025
- While low-hanging, the motivation to combine multiple meta-learning approaches is sound. I mention some caveats below, but I can see how enabling different components of the meta-learner to adapt could accelerate convergence and boost final performance. - In all their experiments, the authors’ method performs the best. The ablations also support their claims.
Conceptual weaknesses: - While it’s tempting to combine existing meta-learning work, a major caveat is not discussed: the more powerful the meta-learner, the higher the risk of meta-overfitting. In other words, the meta-learner risks to overfit to the train task distribution and fail to adapt to new unseen distributions. I wished the authors mentioned this trade-off — and others that arise from designing stronger meta-learner — explicitly and potentially even addressed it directly. - None of th
1. NPBML combines the gradient-based meta-learning methods into a unified end-to-end framework, which meta-learns the key components of learning, i.e., initializations, optimizers, and loss functions, simultaneously. It enables meta-learning to acquire more optimization components and potentially enhances performance. 2. The framework is flexible and general, with many existing gradient-based meta-learning approaches emerging as special cases within NPBML.
1. There is a risk of meta-overfitting, where the model learns too well from the meta-training tasks and fails to generalize to new, unseen tasks. Although the authors mention this issue in the paper and suggest that it can be alleviated using regularization techniques, this introduces many manual choices, which contradicts the goal of automatically learning to learn from tasks. How to prevent meta-overfitting within the NPBML framework should be carefully discussed. 2. Although the authors sta
The paper is clearly presented, motivating the combination of existing components into a new algorithm well.
While the experimental baselines contain a number of bilevel optimization-based meta-learning algorithms that fall into the same paradigm, comparisons to other popular paradigms such as extended pretraining of the backbone (the presented method also pretrains the backbone) and in-context learning / sequence modelling are missing. Such methods [e.g. 1, 2] achieve stronger performance on the few-shot learning tasks evaluated here albeit using larger models. In combination with the large computatio
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsNeural Networks and Applications
MethodsSparse Evolutionary Training