Learning Dynamics of Meta-Learning in Small Model Pretraining

TL;DR

This paper explores meta-learning for small language models, demonstrating faster training, improved multilingual NER performance, and more interpretable training dynamics through a novel integration of MAML with subset-masked pretraining.

Contribution

It introduces a new meta-learning approach combining MAML with subset-masked pretraining for small language models, enhancing training efficiency and interpretability.

Findings

Models reach loss faster than vanilla training.

Improved F1 scores on multilingual NER tasks.

Training dynamics reveal a two-stage diversification and compression process.

Abstract

Large language models are powerful but costly. We ask whether meta-learning can make the pretraining of small language models not only better but also more interpretable. We integrate first-order MAML with subset-masked LM pretraining, producing four LLama-style decoder-only models (11M-570M params), and evaluate it on a fundamental NLP task with many settings and real-world applications. Compared with vanilla training, our model (i) reaches the same loss up to 1.6x sooner, (ii) improves F1 on multilingual Universal NER under equal compute, and (iii) makes the training dynamics easy to read: first the network's representations fan out ("diversify") and later they collapse into a smaller, shared subspace ("compress"). This two-stage shift shows up as a rise-and-fall in both effective-rank curves and attention-head entropy. The same curves pinpoint which layers specialise earliest and which later reconverge, giving a compact, interpretable signature of meta-adaptation. Code, checkpoints and WandB logs are released.