VA-RED$^2$: Video Adaptive Redundancy Reduction

TL;DR

VA-RED$^2$ is an adaptive framework that reduces computational redundancy in video deep learning models by selectively computing features based on input content, achieving significant efficiency gains without performance loss.

Contribution

It introduces an input-dependent, differentiable policy for adaptive feature computation in videos, effectively reducing redundancy and computational cost.

Findings

Achieves 20-40% reduction in FLOPs compared to state-of-the-art methods.

Maintains original model performance despite computational reduction.

Demonstrates effectiveness across multiple datasets and tasks.

Abstract

Performing inference on deep learning models for videos remains a challenge due to the large amount of computational resources required to achieve robust recognition. An inherent property of real-world videos is the high correlation of information across frames which can translate into redundancy in either temporal or spatial feature maps of the models, or both. The type of redundant features depends on the dynamics and type of events in the video: static videos have more temporal redundancy while videos focusing on objects tend to have more channel redundancy. Here we present a redundancy reduction framework, termed VA-RED$^2$, which is input-dependent. Specifically, our VA-RED$^2$ framework uses an input-dependent policy to decide how many features need to be computed for temporal and channel dimensions. To keep the capacity of the original model, after fully computing the necessary features, we reconstruct the remaining redundant features from those using cheap linear operations. We learn the adaptive policy jointly with the network weights in a differentiable way with a shared-weight mechanism, making it highly efficient. Extensive experiments on multiple video datasets and different visual tasks show that our framework achieves $20\% - 40\%$ reduction in computation (FLOPs) when compared to state-of-the-art methods without any performance loss. Project page: http://people.csail.mit.edu/bpan/va-red/.