Intriguing Properties of Randomly Weighted Networks: Generalizing While Learning Next to Nothing

TL;DR

This paper demonstrates that fixing most layers in deep neural networks, especially convolutional ones, often yields comparable performance to fully trained models, revealing surprising robustness and generalization properties.

Contribution

It introduces a method of fixing almost all layers in deep networks, showing that minimal learning can achieve similar results to full training, challenging conventional training paradigms.

Findings

Fixing most network parameters often maintains high performance.

Deep networks exhibit robustness even with limited learned weights.

Potential for creating more robust and efficient neural architectures.

Abstract

Training deep neural networks results in strong learned representations that show good generalization capabilities. In most cases, training involves iterative modification of all weights inside the network via back-propagation. In Extreme Learning Machines, it has been suggested to set the first layer of a network to fixed random values instead of learning it. In this paper, we propose to take this approach a step further and fix almost all layers of a deep convolutional neural network, allowing only a small portion of the weights to be learned. As our experiments show, fixing even the majority of the parameters of the network often results in performance which is on par with the performance of learning all of them. The implications of this intriguing property of deep neural networks are discussed and we suggest ways to harness it to create more robust representations.