Position: We Need An Algorithmic Understanding of Generative AI

TL;DR

This paper introduces AlgEval, a framework for systematically understanding the algorithms learned by large language models, aiming to uncover their internal primitives and improve interpretability and efficiency.

Contribution

It proposes a novel systematic evaluation framework, AlgEval, to analyze and understand the emergent algorithms within LLMs through both top-down hypotheses and bottom-up circuit analysis.

Findings

Case study on emergent search algorithms

Identification of attention and hidden state patterns

Potential for more interpretable and efficient models

Abstract

What algorithms do LLMs actually learn and use to solve problems? Studies addressing this question are sparse, as research priorities are focused on improving performance through scale, leaving a theoretical and empirical gap in understanding emergent algorithms. This position paper proposes AlgEval: a framework for systematic research into the algorithms that LLMs learn and use. AlgEval aims to uncover algorithmic primitives, reflected in latent representations, attention, and inference-time compute, and their algorithmic composition to solve task-specific problems. We highlight potential methodological paths and a case study toward this goal, focusing on emergent search algorithms. Our case study illustrates both the formation of top-down hypotheses about candidate algorithms, and bottom-up tests of these hypotheses via circuit-level analysis of attention patterns and hidden states. The rigorous, systematic evaluation of how LLMs actually solve tasks provides an alternative to resource-intensive scaling, reorienting the field toward a principled understanding of underlying computations. Such algorithmic explanations offer a pathway to human-understandable interpretability, enabling comprehension of the model's internal reasoning performance measures. This can in turn lead to more sample-efficient methods for training and improving performance, as well as novel architectures for end-to-end and multi-agent systems.