Autoregressivity in the Latent Space of a GP-VAE Language Model: An Empirical Ablation Study

TL;DR

This paper empirically analyzes the impact of latent autoregression in GP-VAE language models, showing it enhances long-range stability and structure in the latent space compared to non-autoregressive variants.

Contribution

It provides a systematic ablation study demonstrating the benefits of latent autoregression in GP-VAE models for language processing.

Findings

Latent autoregression improves long-horizon stability.

Removing autoregression degrades latent structure.

Latent autoregression complements token-level autoregressive models.

Abstract

This paper provides an ablation-based analysis of latent autoregression in GP-VAE models, building upon our previous work introducing the architecture. Language models typically rely on an autoregressive factorization over tokens. In contrast, our prior work proposed shifting sequential structure to the latent space through a causal Gaussian process, while using a non-autoregressive decoder. Here, we conduct a systematic ablation study of the role played by latent autoregression. We compare (i) a full GP-VAE model with autoregressive latent dynamics, (ii) a non-autoregressive ablation in which latent variables are independent, and (iii) a standard token-level autoregressive Transformer. Our results show that, within the considered regime (medium-scale corpora and short training contexts), latent autoregression induces latent trajectories that are significantly more compatible with the Gaussian-process prior and exhibit greater long-horizon stability. In contrast, removing autoregression leads to degraded latent structure and unstable long-range behavior. These findings highlight the role of latent autoregression as an effective mechanism for organizing long-range structure, while remaining complementary to token-level autoregressive modeling. They should be interpreted as an empirical analysis of representational structure rather than as a proposal for a new architecture.