Sparsity Moves Computation: How FFN Architecture Reshapes Attention in Small Transformers

TL;DR

This paper investigates how different feedforward network architectures in small Transformers influence the distribution of computational roles between FFN and attention mechanisms, revealing that sparsity can shift computation and affect interpretability.

Contribution

It demonstrates that architectural sparsity in FFNs redistributes computation within Transformers, largely driven by design choices rather than learned routing, and explores implications for interpretability.

Findings

Sparse MoE routing shifts computation from FFN to attention.

Frozen random routing nearly matches learned routing effects.

GLU gating redistributes Fourier structure, affecting interpretability.

Abstract

Architectural choices inside the Transformer feedforward network (FFN) block do not merely affect the block itself; they reshape the computations learned by the rest of the model. We study this effect in one-layer Transformers trained on digit addition with carry, modular arithmetic, and histogram counting. Comparing dense FFNs, gated linear units (GLUs), mixture-of-experts (MoE), and MoE-GLUs, we find that sparse MoE routing can shift computation from FFN to attention, with the strongest ablation-visible effect on carry-based addition. We decompose this redistribution into reduced per-token FFN capacity and sparse partitioning across experts. Critically, frozen random routing nearly matches learned routing, suggesting that redistribution is driven largely by architectural sparsity rather than router-learned specialization. As a secondary finding, GLU-style multiplicative gating rotates task-relevant Fourier structure out of the per-neuron basis and into distributed subspaces, making neuron-level interpretability less informative while preserving structured computation. We validate these conclusions with random-routing, narrow-FFN, and top-2 MoE controls, plus parameter-matching, activation-function, and width-scaling analyses. Together, these results show that local FFN design choices can have nonlocal consequences for Transformer computation.