ResidualDroppath: Enhancing Feature Reuse over Residual Connections

TL;DR

This paper introduces ResidualDroppath, a training method that enhances feature reuse in residual networks by combining droppath and targeted training of dropped layers, leading to improved image classification performance.

Contribution

It proposes a novel training approach that encourages feature reuse in residual networks through iterative droppath and focused training of dropped layers.

Findings

Performance improvements in residual networks for image classification.

Effective promotion of feature reuse via the proposed training method.

Analysis of limitations in vanilla residual connections.

Abstract

Residual connections are one of the most important components in neural network architectures for mitigating the vanishing gradient problem and facilitating the training of much deeper networks. One possible explanation for how residual connections aid deeper network training is by promoting feature reuse. However, we identify and analyze the limitations of feature reuse with vanilla residual connections. To address these limitations, we propose modifications in training methods. Specifically, we provide an additional opportunity for the model to learn feature reuse with residual connections through two types of iterations during training. The first type of iteration involves using droppath, which enforces feature reuse by randomly dropping a subset of layers. The second type of iteration focuses on training the dropped parts of the model while freezing the undropped parts. As a result, the dropped parts learn in a way that encourages feature reuse, as the model relies on the undropped parts with feature reuse in mind. Overall, we demonstrated performance improvements in models with residual connections for image classification in certain cases.