Autoregressive Learning in Joint KL: Sharp Oracle Bounds and Lower Bounds

TL;DR

This paper provides a comprehensive theoretical analysis of autoregressive sequence learning under model misspecification using joint KL divergence, revealing bounds on approximation and estimation errors that depend on sequence length.

Contribution

It establishes the first complete characterization of long-horizon error behavior under joint KL, including matching upper and lower bounds and horizon-free approximation factors.

Findings

Joint KL admits a horizon-free approximation factor, unlike Hellinger-based metrics.

Fundamental lower bound of order H for estimation error, matching upper bounds.

Joint KL guarantees imply policy learning regret bounds similar to existing imitation learning results.

Abstract

We study the fundamental and timely problem of learning long sequences in autoregressive modeling and next-token prediction under model misspecification, measured by the joint Kullback--Leibler (KL) divergence. Our goal is to characterize how the sequence horizon \(H\) affects both approximation and estimation errors in this joint-distribution, sequence-level regime. By establishing matching upper and lower bounds, we provide, to our knowledge, the first complete characterization of long-horizon error behavior under the natural joint KL objective, with improved rates and optimality justification relative to existing work. On the approximation side, we show that joint KL admits a horizon-free approximation factor, in sharp contrast to Hellinger-based analyses that exhibit an \(\Omega(H)\) dependence for computationally efficient methods; this isolates the choice of divergence as the source of approximation amplification. On the estimation side, we prove a fundamental information-theoretic lower bound of order \(\Omega(H)\) that holds for both decomposable policy classes and fully shared policies, matching the \(\widetilde O(H)\) upper bounds achieved by computationally efficient algorithms. Our analysis clarifies the landscape of recent autoregressive learning results by aligning the log-loss training objective, the sequence-level evaluation metric, and the approximation metric {\color{black}through a sharp joint-KL oracle theory}. We further show that these joint-KL guarantees imply policy learning regret bounds at rates matching prior imitation learning literature.