Sequences of Logits Reveal the Low Rank Structure of Language Models

TL;DR

This paper reveals that large language models have an inherent low-rank structure in their logits, which can be exploited for efficient response generation and is supported by both empirical evidence and theoretical analysis.

Contribution

The study introduces a model-agnostic approach to uncover low-rank structures in language models and demonstrates their practical use in response generation.

Findings

Language model logits exhibit low-rank structure across various models.

Low-rank structure enables response generation using unrelated prompts.

Theoretical analysis supports the empirical observations and provides learning guarantees.

Abstract

A major problem in the study of large language models is to understand their inherent low-dimensional structure. We introduce an approach to study the low-dimensional structure of language models at a model-agnostic level: as sequential probabilistic models. We first empirically demonstrate that a wide range of modern language models exhibit low-rank structure: in particular, matrices built from the model's logits for varying sets of prompts and responses have low approximate rank. We then show that this low-rank structure can be leveraged for generation -- in particular, we can generate a response to a target prompt using a linear combination of the model's outputs on unrelated, or even nonsensical prompts. On the theoretical front, we observe that studying the approximate rank of language models in the sense discussed above yields a simple universal abstraction whose theoretical predictions parallel our experiments. We then analyze the representation power of the abstraction and give provable learning guarantees.