LLMs as High-Dimensional Nonlinear Autoregressive Models with Attention: Training, Alignment and Inference

TL;DR

This paper presents a mathematical framework modeling large language models as high-dimensional nonlinear autoregressive systems with attention, clarifying their training, alignment, and inference processes for better understanding and analysis.

Contribution

It introduces an explicit equation-level formulation of LLMs as autoregressive models with attention, unifying various training and inference techniques under a common mathematical framework.

Findings

Self-attention as bilinear-softmax-linear composition

Analysis of alignment behaviors like sycophancy

Insights into inference phenomena such as hallucination and in-context learning

Abstract

Large language models (LLMs) based on transformer architectures are typically described through collections of architectural components and training procedures, obscuring their underlying computational structure. This review article provides a concise mathematical reference for researchers seeking an explicit, equation-level description of LLM training, alignment, and generation. We formulate LLMs as high-dimensional nonlinear autoregressive models with attention-based dependencies. The framework encompasses pretraining via next-token prediction, alignment methods such as reinforcement learning from human feedback (RLHF), direct preference optimization (DPO), rejection sampling fine-tuning (RSFT), and reinforcement learning from verifiable rewards (RLVR), as well as autoregressive generation during inference. Self-attention emerges naturally as a repeated bilinear--softmax--linear composition, yielding highly expressive sequence models. This formulation enables principled analysis of alignment-induced behaviors (including sycophancy), inference-time phenomena (such as hallucination, in-context learning, chain-of-thought prompting, and retrieval-augmented generation), and extensions like continual learning, while serving as a concise reference for interpretation and further theoretical development.