Adaptive Tabu Dropout for Regularization of Deep Neural Network

TL;DR

This paper introduces an adaptive tabu dropout method that dynamically adjusts dropout strategies in deep neural networks, leading to improved regularization and performance over standard dropout techniques.

Contribution

It proposes an adaptive tabu dropout mechanism with automatic tabu tenure selection, enhancing diversification and training effectiveness in deep neural networks.

Findings

Adaptive tabu dropout outperforms standard dropout.

Tabu tenure improves diversification during training.

Method shows significant accuracy gains on benchmark datasets.

Abstract

Dropout is an effective strategy for the regularization of deep neural networks. Applying tabu to the units that have been dropped in the recent epoch and retaining them for training ensures diversification in dropout. In this paper, we improve the Tabu Dropout mechanism for training deep neural networks in two ways. Firstly, we propose to use tabu tenure, or the number of epochs a particular unit will not be dropped. Different tabu tenures provide diversification to boost the training of deep neural networks based on the search landscape. Secondly, we propose an adaptive tabu algorithm that automatically selects the tabu tenure based on the training performances through epochs. On several standard benchmark datasets, the experimental results show that the adaptive tabu dropout and tabu tenure dropout diversify and perform significantly better compared to the standard dropout and basic tabu dropout mechanisms.