Superposition of many models into one

TL;DR

This paper introduces a method to store multiple neural network models within a single set of parameters, allowing individual retrieval and training without interference, effectively utilizing network capacity during training.

Contribution

The authors propose a novel superposition technique enabling multiple models to coexist in one parameter set, expanding the capacity of neural networks for storage and training.

Findings

Large number of models can be stored simultaneously

Models can be trained thousands of steps without interference

Superposition acts as an online form of model compression

Abstract

We present a method for storing multiple models within a single set of parameters. Models can coexist in superposition and still be retrieved individually. In experiments with neural networks, we show that a surprisingly large number of models can be effectively stored within a single parameter instance. Furthermore, each of these models can undergo thousands of training steps without significantly interfering with other models within the superposition. This approach may be viewed as the online complement of compression: rather than reducing the size of a network after training, we make use of the unrealized capacity of a network during training.