Sparse or Dense? A Mechanistic Estimation of Computation Density in Transformer-based LLMs

TL;DR

This paper introduces a method to measure computation density in transformer-based LLMs, revealing that processing is generally dense, varies with input, and correlates with token rarity and context length.

Contribution

It presents a novel density estimator based on mechanistic interpretability, challenging the assumption that LLM computation is sparse and uniform.

Findings

LLMs generally involve dense computation rather than sparse.

Computation density varies depending on input and context.

Rarer tokens and shorter contexts tend to increase density.

Abstract

Transformer-based large language models (LLMs) are comprised of billions of parameters arranged in deep and wide computational graphs. Several studies on LLM efficiency optimization argue that it is possible to prune a significant portion of the parameters, while only marginally impacting performance. This suggests that the computation is not uniformly distributed across the parameters. We introduce here a technique to systematically quantify computation density in LLMs. In particular, we design a density estimator drawing on mechanistic interpretability. We experimentally test our estimator and find that: (1) contrary to what has been often assumed, LLM processing generally involves dense computation; (2) computation density is dynamic, in the sense that models shift between sparse and dense processing regimes depending on the input; (3) per-input density is significantly correlated across LLMs, suggesting that the same inputs trigger either low or high density. Investigating the factors influencing density, we observe that predicting rarer tokens requires higher density, and increasing context length often decreases the density. We believe that our computation density estimator will contribute to a better understanding of the processing at work in LLMs, challenging their symbolic interpretation.