Does Representation Matter? Exploring Intermediate Layers in Large Language Models

TL;DR

This paper investigates the quality of intermediate representations in large language models, revealing that these layers often provide more useful features for tasks than final layers, with implications for model design and training.

Contribution

It introduces a comprehensive analysis of intermediate layer representations in LLMs, highlighting architectural differences and evolution during training using novel metrics.

Findings

Intermediate layers often outperform final layers in representation quality.

Significant architectural differences influence representation evolution.

Bimodal entropy patterns observed in some intermediate layers.

Abstract

Understanding what defines a good representation in large language models (LLMs) is fundamental to both theoretical understanding and practical applications. In this paper, we investigate the quality of intermediate representations in various LLM architectures, including Transformers and State Space Models (SSMs). We find that intermediate layers often yield more informative representations for downstream tasks than the final layers. To measure the representation quality, we adapt and apply a suite of metrics - such as prompt entropy, curvature, and augmentation-invariance - originally proposed in other contexts. Our empirical study reveals significant architectural differences, how representations evolve throughout training, and how factors like input randomness and prompt length affect each layer. Notably, we observe a bimodal pattern in the entropy of some intermediate layers and consider potential explanations tied to training data. Overall, our results illuminate the internal mechanics of LLMs and guide strategies for architectural optimization and training.