Probabilistic Transformers

TL;DR

This paper presents a probabilistic interpretation of Transformers as maximum posterior estimators for Gaussian mixture models, opening avenues for probabilistic extensions and deeper understanding.

Contribution

It introduces a novel probabilistic perspective on Transformers, linking them to Gaussian mixture models and suggesting potential extensions.

Findings

Transformers can be viewed as maximum posterior probability estimators for Gaussian mixtures.

This perspective enables probabilistic extensions of Transformer models.

The approach bridges deep learning and probabilistic modeling.

Abstract

We show that Transformers are Maximum Posterior Probability estimators for Mixtures of Gaussian Models. This brings a probabilistic point of view to Transformers and suggests extensions to other probabilistic cases.