What's Hidden in a Randomly Weighted Neural Network?

TL;DR

This paper reveals that randomly weighted neural networks contain untrained subnetworks capable of high performance, and provides an algorithm to find such subnetworks, challenging the traditional view of training as essential for neural network effectiveness.

Contribution

It introduces the concept of untrained subnetworks within randomly weighted neural networks and presents an algorithm to identify them, demonstrating their competitive performance.

Findings

Untrained subnetworks can match trained network performance.

Larger and deeper random networks contain better untrained subnetworks.

Algorithm effectively finds high-performing subnetworks in random networks.

Abstract

Training a neural network is synonymous with learning the values of the weights. By contrast, we demonstrate that randomly weighted neural networks contain subnetworks which achieve impressive performance without ever training the weight values. Hidden in a randomly weighted Wide ResNet-50 we show that there is a subnetwork (with random weights) that is smaller than, but matches the performance of a ResNet-34 trained on ImageNet. Not only do these "untrained subnetworks" exist, but we provide an algorithm to effectively find them. We empirically show that as randomly weighted neural networks with fixed weights grow wider and deeper, an "untrained subnetwork" approaches a network with learned weights in accuracy. Our code and pretrained models are available at https://github.com/allenai/hidden-networks.