RotationOut as a Regularization Method for Neural Network

TL;DR

RotationOut is a new regularization technique for neural networks that applies random rotations to input vectors, improving generalization across vision and language tasks, and can be combined with Batch Normalization.

Contribution

This paper introduces RotationOut, a novel regularization method that rotates input vectors for neural networks, differing from Dropout, and demonstrates its effectiveness in various tasks.

Findings

RotationOut improves neural network generalization in vision and language tasks.

RotationOut can be combined with Batch Normalization for enhanced performance.

Extensive experiments validate the effectiveness of RotationOut.

Abstract

In this paper, we propose a novel regularization method, RotationOut, for neural networks. Different from Dropout that handles each neuron/channel independently, RotationOut regards its input layer as an entire vector and introduces regularization by randomly rotating the vector. RotationOut can also be used in convolutional layers and recurrent layers with small modifications. We further use a noise analysis method to interpret the difference between RotationOut and Dropout in co-adaptation reduction. Using this method, we also show how to use RotationOut/Dropout together with Batch Normalization. Extensive experiments in vision and language tasks are conducted to show the effectiveness of the proposed method. Codes are available at \url{https://github.com/RotationOut/RotationOut}.