ASNI: Adaptive Structured Noise Injection for shallow and deep neural networks

TL;DR

This paper introduces ASNI, a generalized noise injection method for neural networks that uses structured, adaptive noise instead of independent dropout, improving accuracy and representation quality without extra computational cost.

Contribution

It proposes a novel adaptive structured noise injection technique that generalizes dropout, with theoretical insights and empirical validation showing improved neural network performance.

Findings

Boosts accuracy of feedforward and convolutional networks

Disentangles hidden layer representations

Produces sparser network representations

Abstract

Dropout is a regularisation technique in neural network training where unit activations are randomly set to zero with a given probability \emph{independently}. In this work, we propose a generalisation of dropout and other multiplicative noise injection schemes for shallow and deep neural networks, where the random noise applied to different units is not independent but follows a joint distribution that is either fixed or estimated during training. We provide theoretical insights on why such adaptive structured noise injection (ASNI) may be relevant, and empirically confirm that it helps boost the accuracy of simple feedforward and convolutional neural networks, disentangles the hidden layer representations, and leads to sparser representations. Our proposed method is a straightforward modification of the classical dropout and does not require additional computational overhead.