Dropout with Tabu Strategy for Regularizing Deep Neural Networks

TL;DR

This paper introduces Tabu Dropout, a diversification strategy for dropout in deep neural networks that enhances regularization by promoting more varied network architectures during training.

Contribution

The paper proposes a novel Tabu Dropout method that adds a diversification mechanism to standard dropout without extra parameters, improving neural network regularization.

Findings

Tabu Dropout outperforms standard dropout on MNIST and Fashion-MNIST datasets.

The method is computationally cheap and easy to implement.

It effectively increases the diversity of neural network architectures during training.

Abstract

Dropout has proven to be an effective technique for regularization and preventing the co-adaptation of neurons in deep neural networks (DNN). It randomly drops units with a probability $p$ during the training stage of DNN. Dropout also provides a way of approximately combining exponentially many different neural network architectures efficiently. In this work, we add a diversification strategy into dropout, which aims at generating more different neural network architectures in a proper times of iterations. The dropped units in last forward propagation will be marked. Then the selected units for dropping in the current FP will be kept if they have been marked in the last forward propagation. We only mark the units from the last forward propagation. We call this new technique Tabu Dropout. Tabu Dropout has no extra parameters compared with the standard Dropout and also it is computationally cheap. The experiments conducted on MNIST, Fashion-MNIST datasets show that Tabu Dropout improves the performance of the standard dropout.