Directional Optimization Asymmetry in Transformers: A Synthetic Stress Test

TL;DR

This paper introduces a synthetic benchmark to isolate and analyze the intrinsic directional optimization asymmetry in Transformer models, revealing a fundamental, semantics-free bias that persists even in controlled, language-agnostic settings.

Contribution

The study provides a synthetic, entropy-controlled benchmark to systematically investigate directional learning biases in Transformers, demonstrating an inherent optimization gap independent of linguistic data.

Findings

Transformers exhibit a significant directional optimization gap even in synthetic, language-agnostic tasks.

Pre-training shifts but does not eliminate the directional bias in Transformers.

LoRA encounters a capacity limit on high-entropy inverse mappings, highlighting model constraints.

Abstract

Transformers are theoretically reversal-invariant: their function class does not prefer left-to-right over right-to-left mappings. Yet empirical studies on natural language repeatedly report a "reversal curse," and recent work on temporal asymmetry in LLMs suggests that real-world corpora carry their own arrow of time. This leaves an unresolved question: do directional failures stem from linguistic statistics, or from the architecture itself? We cut through this ambiguity with a fully synthetic, entropy-controlled benchmark designed as a clean-room stress test for directional learning. Using random string mappings with tunable branching factor K, we construct forward tasks with zero conditional entropy and inverse tasks with analytically determined entropy floors. Excess loss above these floors reveals that even scratch-trained GPT-2 models exhibit a strong, reproducible directional optimization gap (e.g., 1.16 nats at K=5), far larger than that of an MLP trained on the same data. Pre-trained initializations shift optimization behavior but do not eliminate this gap, while LoRA encounters a sharp capacity wall on high-entropy inverse mappings. Together, these results isolate a minimal, semantics-free signature of directional friction intrinsic to causal Transformer training-one that persists even when linguistic priors, token frequencies, and corpus-level temporal asymmetries are removed. Our benchmark provides a controlled instrument for dissecting directional biases in modern sequence models and motivates deeper mechanistic study of why inversion remains fundamentally harder for Transformers.