Kalman Linear Attention: Parallel Bayesian Filtering For Efficient Language Modelling and State Tracking

TL;DR

This paper introduces Kalman Linear Attention, a parallelizable Bayesian filtering approach for language modeling that enhances expressivity and state-tracking while maintaining computational efficiency.

Contribution

It reparameterizes Kalman filters for parallel computation and develops KLA, a new attention layer that improves expressivity and uncertainty tracking in sequence modeling.

Findings

KLA matches or outperforms existing SSMs and GLAs.

KLA enables efficient parallel training of probabilistic sequence models.

KLA maintains explicit belief-state uncertainty during language modeling.

Abstract

State-space language models such as Mamba and gated linear attention (GLA) offer efficient alternatives to transformers due to their linear complexity and parallel training, but often lack the expressivity and robust state-tracking needed for complex reasoning. We address these limitations by reframing sequence modelling through a probabilistic lens, using Bayesian filters as a core primitive. While classical filters such as Kalman filters provide principled state estimation and uncertainty tracking, they are typically viewed as inherently sequential. We show that reparameterising the Kalman filter in information form enables its updates to be computed via an associative scan, allowing efficient parallel training. Building on this insight, we introduce the Kalman Linear Attention (KLA) layer, a neural sequence-modelling primitive that performs time-parallel probabilistic inference while maintaining explicit belief-state uncertainty. KLA offers strictly more expressive nonlinear updates and gating than GLA variants while retaining their computational advantages. On language modelling tasks, KLA matches or outperforms modern SSMs and GLAs across representative discrete token-manipulation and state-tracking benchmarks.