# What can we learn from signals and systems in a transformer? Insights for probabilistic modeling and inference architecture

## TL;DR

This paper interprets transformers as probabilistic models, connecting classical filtering theory with modern neural architectures, and provides a fixed-point update formulation for HMMs within this framework.

## Contribution

It introduces a probabilistic interpretation of transformer signals as surrogate conditional measures and describes fixed-point updates for HMMs, bridging classical filtering with modern inference.

## Key findings

- Transformers can be viewed as nonlinear predictors based on probabilistic measures.
- A fixed-point update form is derived for HMMs within the transformer framework.
- The approach bridges classical nonlinear filtering theory with modern neural inference architectures.

## Abstract

In the 1940s, Wiener introduced a linear predictor, where the future prediction is computed by linearly combining the past data. A transformer generalizes this idea: it is a nonlinear predictor where the next-token prediction is computed by nonlinearly combining the past tokens. In this essay, we present a probabilistic model that interprets transformer signals as surrogates of conditional measures, and layer operations as fixed-point updates. An explicit form of the fixed-point update is described for the special case when the probabilistic model is a hidden Markov model (HMM). In part, this paper is in an attempt to bridge the classical nonlinear filtering theory with modern inference architectures.