Continuous Dropout

TL;DR

This paper introduces continuous dropout, a novel regularization technique inspired by neuronal firing patterns, which better mimics brain activity and enhances model performance by reducing feature co-adaptation.

Contribution

The paper proposes continuous dropout, extending traditional binary dropout to be more biologically plausible and effective in preventing feature co-adaptation in deep networks.

Findings

Continuous dropout outperforms binary dropout and DropConnect on multiple datasets.

It better prevents co-adaptation of features, leading to improved test accuracy.

The method aligns more closely with neuronal activation patterns in the brain.

Abstract

Dropout has been proven to be an effective algorithm for training robust deep networks because of its ability to prevent overfitting by avoiding the co-adaptation of feature detectors. Current explanations of dropout include bagging, naive Bayes, regularization, and sex in evolution. According to the activation patterns of neurons in the human brain, when faced with different situations, the firing rates of neurons are random and continuous, not binary as current dropout does. Inspired by this phenomenon, we extend the traditional binary dropout to continuous dropout. On the one hand, continuous dropout is considerably closer to the activation characteristics of neurons in the human brain than traditional binary dropout. On the other hand, we demonstrate that continuous dropout has the property of avoiding the co-adaptation of feature detectors, which suggests that we can extract more independent feature detectors for model averaging in the test stage. We introduce the proposed continuous dropout to a feedforward neural network and comprehensively compare it with binary dropout, adaptive dropout, and DropConnect on MNIST, CIFAR-10, SVHN, NORB, and ILSVRC-12. Thorough experiments demonstrate that our method performs better in preventing the co-adaptation of feature detectors and improves test performance. The code is available at: https://github.com/jasonustc/caffe-multigpu/tree/dropout.