Scaling Bidirectional Spans and Span Violations in Attention Mechanism

TL;DR

This paper introduces an optimization framework for Transformers that decomposes attention gradients into spans and violations, leading to improved training efficiency and performance, especially on larger datasets.

Contribution

It presents a novel gradient decomposition method that enhances Transformer training by focusing on span and violation components, maintaining the forward pass structure.

Findings

Achieved a 0.56% reduction in validation loss on WikiText-2.

The standard attention gradient is shown to be suboptimal.

Selective scaling of gradient components improves learning signals.

Abstract

The canonical $O(N^2)$ Transformer remains the empirical performance frontier in sequence modeling, and its training can be further optimized by addressing geometric inefficiency. We propose an optimization framework that leverages an asymmetric projection to decompose the backward-pass gradients into parallel spans and orthogonal violations, while keeping the canonical forward-pass $QKV$ structure intact. Through consistent experimental validation across various decomposition and projection setups, we provide strong theoretical evidence: the standard attention gradient is suboptimal. We demonstrated that selectively scaling these components, focusing primarily on $0^{th}$ order bidirectional parallel spans, yields the most effective learning signal. On the limited WikiText-2 dataset, and using a crude configuration, this method achieved a $0.56\%$ reduction in validation loss, confirming the framework's fundamental validity and suggesting significant potential gains on larger datasets and deeper training regimes