Perturbative Neural Networks

TL;DR

This paper introduces Perturbative Neural Networks, replacing traditional convolutional layers with a simple perturbation layer that uses additive noise, achieving comparable performance with fewer parameters across various visual datasets.

Contribution

The paper proposes a novel perturbation layer as an alternative to convolutional layers, simplifying architecture while maintaining performance.

Findings

PNNs perform comparably to CNNs on multiple datasets.

PNNs use fewer parameters than standard CNNs.

Perturbation layers are effective replacements for convolutional layers.

Abstract

Convolutional neural networks are witnessing wide adoption in computer vision systems with numerous applications across a range of visual recognition tasks. Much of this progress is fueled through advances in convolutional neural network architectures and learning algorithms even as the basic premise of a convolutional layer has remained unchanged. In this paper, we seek to revisit the convolutional layer that has been the workhorse of state-of-the-art visual recognition models. We introduce a very simple, yet effective, module called a perturbation layer as an alternative to a convolutional layer. The perturbation layer does away with convolution in the traditional sense and instead computes its response as a weighted linear combination of non-linearly activated additive noise perturbed inputs. We demonstrate both analytically and empirically that this perturbation layer can be an effective replacement for a standard convolutional layer. Empirically, deep neural networks with perturbation layers, called Perturbative Neural Networks (PNNs), in lieu of convolutional layers perform comparably with standard CNNs on a range of visual datasets (MNIST, CIFAR-10, PASCAL VOC, and ImageNet) with fewer parameters.