Transformer-based Causal Language Models Perform Clustering

TL;DR

This paper investigates how Transformer-based causal language models learn to follow instructions by analyzing their internal clustering mechanisms, revealing insights into their dynamic learning process and implications for pre-training and alignment.

Contribution

The study introduces a simplified instruction-following task and demonstrates that clustering in hidden space underpins the model's ability to generalize and follow instructions effectively.

Findings

Models learn task-specific info through dynamic clustering.

Clustering aids in handling unseen instances.

Results validated in realistic settings.

Abstract

Even though large language models (LLMs) have demonstrated remarkable capability in solving various natural language tasks, the capability of an LLM to follow human instructions is still a concern. Recent works have shown great improvements in the instruction-following capability via additional training for instruction-following tasks. However, the mechanisms responsible for effective instruction-following capabilities remain inadequately understood. Here, we introduce a simplified instruction-following task and use synthetic datasets to analyze a Transformer-based causal language model. Our findings suggest that the model learns task-specific information by clustering data within its hidden space, with this clustering process evolving dynamically during learning. We also demonstrate how this phenomenon assists the model in handling unseen instances, and validate our results in a more realistic setting. Furthermore, we present inspired applications regarding pre-training and alignment.