Regularizing Neural Networks by Stochastically Training Layer Ensembles

TL;DR

This paper introduces STE layers, a stochastic ensemble training method that improves neural network regularization by explicitly averaging multiple weight matrices, leading to better image classification performance without extra test-time cost.

Contribution

The paper proposes STE layers, a novel stochastic ensemble training approach that enhances regularization and model averaging in neural networks.

Findings

Consistent improvement on image classification tasks.

No additional computational cost during testing.

Enhanced regularization compared to traditional methods.

Abstract

Dropout and similar stochastic neural network regularization methods are often interpreted as implicitly averaging over a large ensemble of models. We propose STE (stochastically trained ensemble) layers, which enhance the averaging properties of such methods by training an ensemble of weight matrices with stochastic regularization while explicitly averaging outputs. This provides stronger regularization with no additional computational cost at test time. We show consistent improvement on various image classification tasks using standard network topologies.