PRISM: Parallel Residual Iterative Sequence Model

TL;DR

PRISM is a novel parallelizable sequence model that combines the expressivity of iterative refinement with the efficiency of linear models, achieving high throughput without sacrificing performance.

Contribution

PRISM introduces a solver-inspired inductive bias and a two-stage proxy architecture to enable parallel multi-step refinement in sequence modeling.

Findings

Achieves Rank-$L$ accumulation, expanding the update manifold.

Attains performance comparable to explicit optimization methods.

Provides 174x higher throughput than traditional iterative methods.

Abstract

Generative sequence modeling faces a fundamental tension between the expressivity of Transformers and the efficiency of linear sequence models. Existing efficient architectures are theoretically bounded by shallow, single-step linear updates, while powerful iterative methods like Test-Time Training (TTT) break hardware parallelism due to state-dependent gradients. We propose PRISM (Parallel Residual Iterative Sequence Model) to resolve this tension. PRISM introduces a solver-inspired inductive bias that captures key structural properties of multi-step refinement in a parallelizable form. We employ a Write-Forget Decoupling strategy that isolates non-linearity within the injection operator. To bypass the serial dependency of explicit solvers, PRISM utilizes a two-stage proxy architecture: a short-convolution anchors the initial residual using local history energy, while a learned predictor estimates the refinement updates directly from the input. This design distills structural patterns associated with iterative correction into a parallelizable feedforward operator. Theoretically, we prove that this formulation achieves Rank-$L$ accumulation, structurally expanding the update manifold beyond the single-step Rank-$1$ bottleneck. Empirically, it achieves comparable performance to explicit optimization methods while achieving 174x higher throughput.