Stochastic Backpropagation: A Memory Efficient Strategy for Training Video Models

TL;DR

This paper introduces Stochastic Backpropagation (SBP), a memory-efficient training method for video neural networks that reduces GPU memory usage and speeds up training with minimal accuracy loss by randomly dropping backward paths during training.

Contribution

The paper presents SBP, a novel approach that significantly reduces memory consumption in training video models by selectively dropping backward paths, enabling more efficient training.

Findings

Up to 80% GPU memory savings

10% training speedup

Less than 1% accuracy drop

Abstract

We propose a memory efficient method, named Stochastic Backpropagation (SBP), for training deep neural networks on videos. It is based on the finding that gradients from incomplete execution for backpropagation can still effectively train the models with minimal accuracy loss, which attributes to the high redundancy of video. SBP keeps all forward paths but randomly and independently removes the backward paths for each network layer in each training step. It reduces the GPU memory cost by eliminating the need to cache activation values corresponding to the dropped backward paths, whose amount can be controlled by an adjustable keep-ratio. Experiments show that SBP can be applied to a wide range of models for video tasks, leading to up to 80.0% GPU memory saving and 10% training speedup with less than 1% accuracy drop on action recognition and temporal action detection.