Task Diversity Shortens the ICL Plateau

TL;DR

This paper shows that training language models on diverse in-context learning tasks shortens the long learning plateaus, making each task easier to learn and potentially explaining the success of large-scale models.

Contribution

It reveals that task diversity during training reduces loss plateaus in ICL, contradicting prior intuition about complexity and facilitating easier learning.

Findings

Diverse ICL tasks shorten the loss plateau duration.

Training on multiple tasks improves learning efficiency.

Task diversity may contribute to large-scale model success.

Abstract

In-context learning (ICL) describes a language model's ability to generate outputs based on a set of input demonstrations and a subsequent query. To understand this remarkable capability, researchers have studied simplified, stylized models. These studies have consistently observed long loss plateaus, during which models exhibit minimal improvement, followed by a sudden, rapid surge of learning. In this work, we reveal that training on multiple diverse ICL tasks simultaneously shortens the loss plateaus, making each task easier to learn. This finding is surprising as it contradicts the natural intuition that the combined complexity of multiple ICL tasks would lengthen the learning process, not shorten it. Our result suggests that the recent success in large-scale training of language models may be attributed not only to the richness of the data at scale but also to the easier optimization (training) induced by the diversity of natural language training data.