Slot Machines: Discovering Winning Combinations of Random Weights in Neural Networks

TL;DR

This paper introduces a novel neural network approach using fixed random weights selected from a small set, achieving competitive performance without traditional weight training.

Contribution

It presents a method where fixed random weights are selected via backpropagation to match or surpass trained networks, a significant departure from conventional training.

Findings

Achieves 91% accuracy on CIFAR-10 with fixed random weights.

Attains 98.2% accuracy on MNIST using only random weights.

Few random values per connection suffice for high performance.

Abstract

In contrast to traditional weight optimization in a continuous space, we demonstrate the existence of effective random networks whose weights are never updated. By selecting a weight among a fixed set of random values for each individual connection, our method uncovers combinations of random weights that match the performance of traditionally-trained networks of the same capacity. We refer to our networks as "slot machines" where each reel (connection) contains a fixed set of symbols (random values). Our backpropagation algorithm "spins" the reels to seek "winning" combinations, i.e., selections of random weight values that minimize the given loss. Quite surprisingly, we find that allocating just a few random values to each connection (e.g., 8 values per connection) yields highly competitive combinations despite being dramatically more constrained compared to traditionally learned weights. Moreover, finetuning these combinations often improves performance over the trained baselines. A randomly initialized VGG-19 with 8 values per connection contains a combination that achieves 91% test accuracy on CIFAR-10. Our method also achieves an impressive performance of 98.2% on MNIST for neural networks containing only random weights.