Variational Dropout Sparsifies Deep Neural Networks

TL;DR

This paper extends Variational Dropout for deep neural networks, enabling unbounded dropout rates and resulting in highly sparse models with significantly fewer parameters and minimal accuracy loss.

Contribution

It introduces a method for unbounded dropout rates, reduces gradient variance, and demonstrates extreme sparsity in neural networks.

Findings

Parameter reduction up to 280 times on LeNet

Parameter reduction up to 68 times on VGG-like networks

Minimal accuracy decrease despite sparsity

Abstract

We explore a recently proposed Variational Dropout technique that provided an elegant Bayesian interpretation to Gaussian Dropout. We extend Variational Dropout to the case when dropout rates are unbounded, propose a way to reduce the variance of the gradient estimator and report first experimental results with individual dropout rates per weight. Interestingly, it leads to extremely sparse solutions both in fully-connected and convolutional layers. This effect is similar to automatic relevance determination effect in empirical Bayes but has a number of advantages. We reduce the number of parameters up to 280 times on LeNet architectures and up to 68 times on VGG-like networks with a negligible decrease of accuracy.