Language Modelling as a Multi-Task Problem

TL;DR

This paper explores language modelling as a multi-task problem, analyzing how models learn linguistic concepts like Negative Polarity Items and revealing emergent multi-task behavior that benefits linguistics and interpretability research.

Contribution

It introduces a multi-task perspective to language modelling based on linguistic theory, demonstrating natural emergence of multi-task learning in language models.

Findings

Language models naturally exhibit multi-task learning behavior.

Models effectively learn the linguistic concept of Negative Polarity Items.

Insights support interdisciplinary research in linguistics and interpretability.

Abstract

In this paper, we propose to study language modelling as a multi-task problem, bringing together three strands of research: multi-task learning, linguistics, and interpretability. Based on hypotheses derived from linguistic theory, we investigate whether language models adhere to learning principles of multi-task learning during training. To showcase the idea, we analyse the generalisation behaviour of language models as they learn the linguistic concept of Negative Polarity Items (NPIs). Our experiments demonstrate that a multi-task setting naturally emerges within the objective of the more general task of language modelling.We argue that this insight is valuable for multi-task learning, linguistics and interpretability research and can lead to exciting new findings in all three domains.