Skip-Convolutions for Efficient Video Processing

TL;DR

Skip-Convolutions are a novel method that significantly reduce video processing computations by selectively skipping redundant residuals, maintaining accuracy while improving efficiency across multiple architectures.

Contribution

The paper introduces Skip-Convolutions, a new approach that leverages residual-based gating to efficiently process videos, achieving 3-4x speedups without accuracy loss.

Findings

Reduce computational cost by 3-4x in state-of-the-art architectures

Maintain accuracy while exploiting temporal redundancies

Set new state-of-the-art in video efficiency methods

Abstract

We propose Skip-Convolutions to leverage the large amount of redundancies in video streams and save computations. Each video is represented as a series of changes across frames and network activations, denoted as residuals. We reformulate standard convolution to be efficiently computed on residual frames: each layer is coupled with a binary gate deciding whether a residual is important to the model prediction,~\eg foreground regions, or it can be safely skipped, e.g. background regions. These gates can either be implemented as an efficient network trained jointly with convolution kernels, or can simply skip the residuals based on their magnitude. Gating functions can also incorporate block-wise sparsity structures, as required for efficient implementation on hardware platforms. By replacing all convolutions with Skip-Convolutions in two state-of-the-art architectures, namely EfficientDet and HRNet, we reduce their computational cost consistently by a factor of 3~4x for two different tasks, without any accuracy drop. Extensive comparisons with existing model compression, as well as image and video efficiency methods demonstrate that Skip-Convolutions set a new state-of-the-art by effectively exploiting the temporal redundancies in videos.