Randomly Initialized Networks Can Learn from Peer-to-Peer Consensus

TL;DR

This paper demonstrates that even minimal, randomly initialized networks can improve their representations through peer-to-peer self-distillation, without complex mechanisms or pretext tasks.

Contribution

It isolates the effect of self-distillation in learning dynamics by training simple, randomly initialized networks, revealing their capacity to learn useful representations.

Findings

Randomly initialized networks can learn improved representations via self-distillation.

The effect varies with different hyperparameters.

Minimal setups can outperform random baselines on downstream tasks.

Abstract

In self-supervised learning, self-distilled methods have shown impressive performance, learning representations useful for downstream tasks and even displaying emergent properties. However, state-of-the-art methods usually rely on ensembles of complex mechanisms, with many design choices that are empirically motivated and not well understood. In this work, we explore the role of self-distillation within learning dynamics. Specifically, we isolate the effect of self-distillation by training a group of randomly initialized networks, removing all other common components such as projectors, predictors, and even pretext tasks. Our findings show that even this minimal setup can lead to learned representations with non-trivial improvements over a random baseline on downstream tasks. We also demonstrate how this effect varies with different hyperparameters and present a short analysis of what is being learned by the models under this setup.