
TL;DR
This paper demonstrates that the order of training data significantly influences neural network learning, enabling models to generalize from minimal data by encoding information in the sequence structure, with implications for efficiency and security.
Contribution
It reveals that data ordering can encode information in neural networks, achieving high accuracy with minimal data and challenging assumptions about training complexity.
Findings
Order-based training strategies reach 99.5% accuracy with only 0.3% of data.
Adversarial ordering prevents learning entirely.
Models encode information in Fourier representations linked to data order.
Abstract
In a controlled experiment on modular arithmetic (), varying only example ordering while holding all else constant, two fixed-ordering strategies achieve 99.5\% test accuracy by epochs 487 and 659 respectively from a training set comprising 0.3\% of the input space, well below established sample complexity lower bounds for this task under IID ordering. The IID baseline achieves 0.30\% after 5{,}000 epochs from identical data. An adversarially structured ordering suppresses learning entirely. The generalizing model reliably constructs a Fourier representation whose fundamental frequency is the Fourier dual of the ordering structure, encoding information present in no individual training example, with the same fundamental emerging across all seeds tested regardless of initialization or training set composition. We discuss implications for training efficiency, the…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsMachine Learning and Algorithms · Adversarial Robustness in Machine Learning · Ferroelectric and Negative Capacitance Devices
