Meta-Learning a Dynamical Language Model

TL;DR

This paper introduces a meta-learning approach for language models that combines short-term hidden states with medium-term dynamical weights, enabling continuous online adaptation through gradient-based meta-learning.

Contribution

It presents a novel online learning-to-learn framework that trains a meta-learner to dynamically update language model weights for improved adaptability.

Findings

Enhanced language modeling through dynamic weight updates

Effective integration of hidden states and dynamical weights

Demonstrated continuous online learning capability

Abstract

We consider the task of word-level language modeling and study the possibility of combining hidden-states-based short-term representations with medium-term representations encoded in dynamical weights of a language model. Our work extends recent experiments on language models with dynamically evolving weights by casting the language modeling problem into an online learning-to-learn framework in which a meta-learner is trained by gradient-descent to continuously update a language model weights.