Rethinking LLM Advancement: Compute-Dependent and Independent Paths to Progress

TL;DR

This paper introduces a framework to distinguish between compute-dependent and compute-independent innovations in LLM development, showing that algorithmic improvements can still advance capabilities despite hardware restrictions.

Contribution

The study proposes a novel framework for classifying LLM innovations and demonstrates its effectiveness through experimental validation with nanoGPT models.

Findings

Compute-independent innovations significantly improve performance across scales.

Compute-dependent innovations benefit mainly at larger scales, with mixed effects at smaller scales.

Hardware restrictions alone are insufficient to halt all AI capability progress.

Abstract

Regulatory efforts to govern large language model (LLM) development have predominantly focused on restricting access to high-performance computational resources. This study evaluates the efficacy of such measures by examining whether LLM capabilities can advance through algorithmic innovation in compute-constrained environments. We propose a novel framework distinguishing compute-dependent innovations--which yield disproportionate benefits at high compute--from compute-independent innovations, which improve efficiency across compute scales. The impact is quantified using Compute-Equivalent Gain (CEG). Experimental validation with nanoGPT models confirms that compute-independent advancements yield significant performance gains (e.g., with combined CEG up to $3.5\times$) across the tested scales. In contrast, compute-dependent advancements were detrimental to performance at smaller experimental scales, but showed improved CEG (on par with the baseline) as model size increased, a trend consistent with their definition of yielding primary benefits at higher compute. Crucially, these findings indicate that restrictions on computational hardware, while potentially slowing LLM progress, are insufficient to prevent all capability gains driven by algorithmic advancements. We argue that effective AI oversight must therefore incorporate mechanisms for understanding, anticipating, and potentially guiding algorithmic research, moving beyond a singular focus on hardware. The proposed framework also serves as an analytical tool for forecasting AI progress.