R-Block: Regularized Block of Dropout for convolutional networks

TL;DR

This paper introduces R-Block, a regularized dropout method for convolutional networks that uses mutual learning between sub-models to improve regularization effectiveness and consistency during training and inference.

Contribution

R-Block applies a mutual learning strategy to structured dropout in convolutional layers, enhancing regularization and model performance.

Findings

R-Block outperforms existing structured dropout methods.

Mutual learning improves training consistency.

Constructing sub-models effectively enhances regularization.

Abstract

Dropout as a regularization technique is widely used in fully connected layers while is less effective in convolutional layers. Therefore more structured forms of dropout have been proposed to regularize convolutional networks. The disadvantage of these methods is that the randomness introduced causes inconsistency between training and inference. In this paper, we apply a mutual learning training strategy for convolutional layer regularization, namely R-Block, which forces two outputs of the generated difference maximizing sub models to be consistent with each other. Concretely, R-Block minimizes the losses between the output distributions of two sub models with different drop regions for each sample in the training dataset. We design two approaches to construct such sub models. Our experiments demonstrate that R-Block achieves better performance than other existing structured dropout variants. We also demonstrate that our approaches to construct sub models outperforms others.