Quantal synaptic dilution enhances sparse encoding and dropout regularisation in deep networks

TL;DR

Quantal Synaptic Dilution (QSD) is a biologically inspired dropout method that improves regularisation and sparse encoding in deep networks, outperforming standard dropout across various architectures and tasks.

Contribution

Introduces QSD, a biologically plausible dropout technique based on synaptic quantal properties, enhancing deep network regularisation and generalisation.

Findings

QSD outperforms standard dropout in multilayer perceptrons.

QSD improves generalisation in convolutional networks for vision tasks.

QSD surpasses standard dropout in recurrent networks for language tasks.

Abstract

Dropout is a technique that silences the activity of units stochastically while training deep networks to reduce overfitting. Here we introduce Quantal Synaptic Dilution (QSD), a biologically plausible model of dropout regularisation based on the quantal properties of neuronal synapses, that incorporates heterogeneities in response magnitudes and release probabilities for vesicular quanta. QSD outperforms standard dropout in ReLU multilayer perceptrons, with enhanced sparse encoding at test time when dropout masks are replaced with identity functions, without shifts in trainable weight or bias distributions. For convolutional networks, the method also improves generalisation in computer vision tasks with and without inclusion of additional forms of regularisation. QSD also outperforms standard dropout in recurrent networks for language modelling and sentiment analysis. An advantage of QSD over many variations of dropout is that it can be implemented generally in all conventional deep networks where standard dropout is applicable.