More Is Less: Learning Efficient Video Representations by Big-Little Network and Depthwise Temporal Aggregation

TL;DR

This paper introduces a lightweight, efficient video action recognition architecture combining a deep low-resolution subnet with a compact high-resolution subnet, significantly reducing computational costs while maintaining or improving accuracy.

Contribution

The paper proposes a novel Big-Little Network architecture with depthwise temporal aggregation, enabling high efficiency and accuracy in video recognition with reduced resource requirements.

Findings

Achieves 3-4x reduction in FLOPs compared to baseline.

Uses 2x less memory while maintaining performance.

Performs well on Kinetics, Something-Something, and Moments-in-Time.

Abstract

Current state-of-the-art models for video action recognition are mostly based on expensive 3D ConvNets. This results in a need for large GPU clusters to train and evaluate such architectures. To address this problem, we present a lightweight and memory-friendly architecture for action recognition that performs on par with or better than current architectures by using only a fraction of resources. The proposed architecture is based on a combination of a deep subnet operating on low-resolution frames with a compact subnet operating on high-resolution frames, allowing for high efficiency and accuracy at the same time. We demonstrate that our approach achieves a reduction by $3\sim4$ times in FLOPs and $\sim2$ times in memory usage compared to the baseline. This enables training deeper models with more input frames under the same computational budget. To further obviate the need for large-scale 3D convolutions, a temporal aggregation module is proposed to model temporal dependencies in a video at very small additional computational costs. Our models achieve strong performance on several action recognition benchmarks including Kinetics, Something-Something and Moments-in-time. The code and models are available at https://github.com/IBM/bLVNet-TAM.