ChannelDropBack: Forward-Consistent Stochastic Regularization for Deep Networks

TL;DR

ChannelDropBack introduces a novel stochastic regularization method that applies randomness during backpropagation only, enhancing deep network training without altering architecture or inference behavior.

Contribution

It presents a simple, backward-only stochastic regularization technique compatible with any network architecture and layer type, maintaining the same model for training and deployment.

Findings

Improves accuracy on ImageNet and ViT datasets

Seamless integration without architecture modifications

Effective across various network topologies

Abstract

Incorporating stochasticity into the training process of deep convolutional networks is a widely used technique to reduce overfitting and improve regularization. Existing techniques often require modifying the architecture of the network by adding specialized layers, are effective only to specific network topologies or types of layers - linear or convolutional, and result in a trained model that is different from the deployed one. We present ChannelDropBack, a simple stochastic regularization approach that introduces randomness only into the backward information flow, leaving the forward pass intact. ChannelDropBack randomly selects a subset of channels within the network during the backpropagation step and applies weight updates only to them. As a consequence, it allows for seamless integration into the training process of any model and layers without the need to change its architecture, making it applicable to various network topologies, and the exact same network is deployed during training and inference. Experimental evaluations validate the effectiveness of our approach, demonstrating improved accuracy on popular datasets and models, including ImageNet and ViT. Code is available at \url{https://github.com/neiterman21/ChannelDropBack.git}.