Let's Agree to Agree: Neural Networks Share Classification Order on Real Datasets

TL;DR

Deep neural networks tend to learn data examples in a similar order across different models and datasets, revealing insights into how they discover structure in natural data.

Contribution

This paper demonstrates that neural networks learn examples in a consistent order across models and datasets, highlighting a shared learning pattern and its dependence on architecture and data structure.

Findings

Neural networks learn training and test data in similar order.

This pattern persists across different architectures and datasets.

Synthetic datasets can disrupt this learning order.

Abstract

We report a series of robust empirical observations, demonstrating that deep Neural Networks learn the examples in both the training and test sets in a similar order. This phenomenon is observed in all the commonly used benchmarks we evaluated, including many image classification benchmarks, and one text classification benchmark. While this phenomenon is strongest for models of the same architecture, it also crosses architectural boundaries -- models of different architectures start by learning the same examples, after which the more powerful model may continue to learn additional examples. We further show that this pattern of results reflects the interplay between the way neural networks learn benchmark datasets. Thus, when fixing the architecture, we show synthetic datasets where this pattern ceases to exist. When fixing the dataset, we show that other learning paradigms may learn the data in a different order. We hypothesize that our results reflect how neural networks discover structure in natural datasets.