ASAP: Amortized Doubly-Stochastic Attention via Sliced Dual Projection

TL;DR

ASAP is a method that trains doubly-stochastic attention with Sinkhorn and then replaces iterative scaling at inference with a fixed operator, achieving efficiency and competitive performance.

Contribution

It introduces a train-then-compile approach that replaces online iteration with a parametric map, reducing inference cost while maintaining accuracy.

Findings

ASAP is 5.3 times faster than Sinkhorn with similar accuracy.

It recovers most teacher performance without retraining in downstream tasks.

Maintains high performance across language and vision benchmarks.

Abstract

Doubly-stochastic attention has emerged as a transport-based alternative to row-softmax attention, with recent Transformer variants using it to reduce attention sinks and rank collapse while improving performance. In this family, the standard approach is Sinkhorn scaling, which trains more efficiently but still repeats matrix scaling in every inference forward pass. Sliced-transport attention removes the online iteration, but its soft sorting approximation materializes dense tensors for each slice, requiring substantially more training resources than Sinkhorn attention. We introduce ASAP: Amortized Doubly-Stochastic Attention via Sliced Dual Projection, a train-then-compile method that trains the doubly-stochastic layer with Sinkhorn, then replaces the iterative scaling loop at inference with a fixed sliced-dual operator. It learns a lightweight parametric map from exact one-dimensional Kantorovich potentials to the Sinkhorn query-side dual, then reconstructs the attention plan with a two-sided entropic c-transform. Across language and vision benchmarks, ASAP keeps the cheaper training setup and remains highly competitive with recent baselines. In the main frozen-layer benchmark, ASAP is 5.3 faster than the trained Sinkhorn teacher while matching its accuracy; in downstream replacements, ASAP recovers most of the teacher performance without any retraining.