Dropout Inference with Non-Uniform Weight Scaling

TL;DR

This paper investigates non-uniform weight scaling during dropout inference, showing it can better approximate model outputs when submodels vary in bias, improving over traditional uniform scaling methods.

Contribution

It introduces a non-uniform weight scaling approach for dropout inference, addressing limitations of uniform scaling in scenarios with high-bias submodels.

Findings

Non-uniform scaling improves inference accuracy in certain dropout scenarios.

Traditional uniform scaling may be suboptimal when submodels have varying biases.

The proposed method offers a better approximation in specific model configurations.

Abstract

Dropout as regularization has been used extensively to prevent overfitting for training neural networks. During training, units and their connections are randomly dropped, which could be considered as sampling many different submodels from the original model. At test time, weight scaling and Monte Carlo approximation are two widely applied approaches to approximate the outputs. Both approaches work well practically when all submodels are low-bias complex learners. However, in this work, we demonstrate scenarios where some submodels behave closer to high-bias models and a non-uniform weight scaling is a better approximation for inference.