LayerDropBack: A Universally Applicable Approach for Accelerating Training of Deep Networks

TL;DR

LayerDropBack (LDB) is a simple method that accelerates training of deep networks by introducing randomness only in the backward pass, applicable across various architectures without modifying the network structure.

Contribution

LDB is a novel, architecture-agnostic training acceleration technique that maintains model integrity and improves training efficiency across diverse deep learning models.

Findings

Achieves 16.93% to 23.97% reduction in training time.

Maintains or improves model accuracy across multiple architectures.

Applicable to various network topologies without architectural modifications.

Abstract

Training very deep convolutional networks is challenging, requiring significant computational resources and time. Existing acceleration methods often depend on specific architectures or require network modifications. We introduce LayerDropBack (LDB), a simple yet effective method to accelerate training across a wide range of deep networks. LDB introduces randomness only in the backward pass, maintaining the integrity of the forward pass, guaranteeing that the same network is used during both training and inference. LDB can be seamlessly integrated into the training process of any model without altering its architecture, making it suitable for various network topologies. Our extensive experiments across multiple architectures (ViT, Swin Transformer, EfficientNet, DLA) and datasets (CIFAR-100, ImageNet) show significant training time reductions of 16.93\% to 23.97\%, while preserving or even enhancing model accuracy. Code is available at \url{https://github.com/neiterman21/LDB}.