Adversarial Dropout for Supervised and Semi-supervised Learning

TL;DR

This paper introduces adversarial dropout, a method that identifies critical dropout patterns to improve neural network generalization in supervised and semi-supervised learning, outperforming standard dropout on MNIST and CIFAR-10.

Contribution

The paper proposes adversarial dropout, a novel technique that enhances neural network training by maximizing output divergence, leading to better generalization in supervised and semi-supervised tasks.

Findings

Adversarial dropout improves generalization on MNIST and CIFAR-10.

It increases network sparsity more than standard dropout.

Training on adversarially reconfigured sub-networks yields better performance.

Abstract

Recently, the training with adversarial examples, which are generated by adding a small but worst-case perturbation on input examples, has been proved to improve generalization performance of neural networks. In contrast to the individually biased inputs to enhance the generality, this paper introduces adversarial dropout, which is a minimal set of dropouts that maximize the divergence between the outputs from the network with the dropouts and the training supervisions. The identified adversarial dropout are used to reconfigure the neural network to train, and we demonstrated that training on the reconfigured sub-network improves the generalization performance of supervised and semi-supervised learning tasks on MNIST and CIFAR-10. We analyzed the trained model to reason the performance improvement, and we found that adversarial dropout increases the sparsity of neural networks more than the standard dropout does.