The Order Is The Message

TL;DR

This paper demonstrates that the order of training data significantly influences neural network learning, enabling models to generalize from minimal data by encoding information in the sequence structure, with implications for efficiency and security.

Contribution

It reveals that data ordering can encode information in neural networks, achieving high accuracy with minimal data and challenging assumptions about training complexity.

Findings

Order-based training strategies reach 99.5% accuracy with only 0.3% of data.

Adversarial ordering prevents learning entirely.

Models encode information in Fourier representations linked to data order.

Abstract

In a controlled experiment on modular arithmetic ($p = 9973$), varying only example ordering while holding all else constant, two fixed-ordering strategies achieve 99.5\% test accuracy by epochs 487 and 659 respectively from a training set comprising 0.3\% of the input space, well below established sample complexity lower bounds for this task under IID ordering. The IID baseline achieves 0.30\% after 5{,}000 epochs from identical data. An adversarially structured ordering suppresses learning entirely. The generalizing model reliably constructs a Fourier representation whose fundamental frequency is the Fourier dual of the ordering structure, encoding information present in no individual training example, with the same fundamental emerging across all seeds tested regardless of initialization or training set composition. We discuss implications for training efficiency, the reinterpretation of grokking, and the safety risks of a channel that evades all content-level auditing.