Understanding Dropout as an Optimization Trick

TL;DR

This paper offers a new perspective on dropout as an optimization method that pushes inputs into activation saturation zones, and introduces GAAF, a technique to enhance gradient flow, improving neural network robustness and performance.

Contribution

It provides a novel explanation of dropout's effectiveness and proposes GAAF, a new activation function technique that accelerates gradients in saturation areas.

Findings

GAAF improves image classification accuracy.

Dropout's effect is partly due to pushing inputs into saturation zones.

GAAF enhances robustness by enabling better gradient flow.

Abstract

As one of standard approaches to train deep neural networks, dropout has been applied to regularize large models to avoid overfitting, and the improvement in performance by dropout has been explained as avoiding co-adaptation between nodes. However, when correlations between nodes are compared after training the networks with or without dropout, one question arises if co-adaptation avoidance explains the dropout effect completely. In this paper, we propose an additional explanation of why dropout works and propose a new technique to design better activation functions. First, we show that dropout can be explained as an optimization technique to push the input towards the saturation area of nonlinear activation function by accelerating gradient information flowing even in the saturation area in backpropagation. Based on this explanation, we propose a new technique for activation functions, {\em gradient acceleration in activation function (GAAF)}, that accelerates gradients to flow even in the saturation area. Then, input to the activation function can climb onto the saturation area which makes the network more robust because the model converges on a flat region. Experiment results support our explanation of dropout and confirm that the proposed GAAF technique improves image classification performance with expected properties.