Latent-Autoregressive GP-VAE Language Model

TL;DR

This paper introduces a novel Latent-Autoregressive GP-VAE language model that transfers sequential dynamics to a continuous latent space, enabling stable training and consistent sampling behaviors, with implications for probabilistic language modeling.

Contribution

It presents a fully probabilistic framework combining Gaussian Processes with VAEs for language modeling, emphasizing latent space geometry over explicit neural autoregression.

Findings

Model trains stably within a proof-of-concept framework

Sequential and parallel sampling variants behave consistently

Latent space geometry supports temporal structure in language modeling

Abstract

We investigate a fully Latent AutoRegressive scheme based on a Gaussian Process (GP) integrated into a Variational Autoencoder (VAE). In this setting, sequential dynamics are transferred from the observation space to a continuous latent space, while linguistic generation remains parallel through a non-autoregressive decoder. We present a complete methodological formulation, including a causal GP prior, a structured amortized posterior, and a training protocol based on a regularized ELBO. Empirical evaluation, conducted within a deliberately constrained proof-of-concept (POC) framework, shows that the model can be trained stably and that the sequential and parallel sampling variants exhibit consistent behavior. Overall, the results suggest that part of the temporal structure in a language model can be supported by the probabilistic geometry of the latent space rather than by explicit neural operations.