Smooth regularization for efficient video recognition

TL;DR

This paper introduces a smooth regularization method for video recognition that enhances lightweight models by modeling temporal coherence as a Gaussian Random Walk, leading to significant accuracy improvements.

Contribution

The paper presents a novel smooth regularization technique based on Gaussian Random Walks that improves temporal modeling in lightweight video recognition architectures.

Findings

Achieves 3.8% to 6.4% accuracy improvements on Kinetics-600.

State-of-the-art results for MoViNets with 3.8% to 6.1% gains.

MobileNetV3 and MoViNets-Stream see 4.9% to 6.4% accuracy boosts.

Abstract

We propose a smooth regularization technique that instills a strong temporal inductive bias in video recognition models, particularly benefiting lightweight architectures. Our method encourages smoothness in the intermediate-layer embeddings of consecutive frames by modeling their changes as a Gaussian Random Walk (GRW). This penalizes abrupt representational shifts, thereby promoting low-acceleration solutions that better align with the natural temporal coherence inherent in videos. By leveraging this enforced smoothness, lightweight models can more effectively capture complex temporal dynamics. Applied to such models, our technique yields a 3.8% to 6.4% accuracy improvement on Kinetics-600. Notably, the MoViNets model family trained with our smooth regularization improves the current state of the art by 3.8% to 6.1% within their respective FLOP constraints, while MobileNetV3 and the MoViNets-Stream family achieve gains of 4.9% to 6.4% over prior state-of-the-art models with comparable memory footprints. Our code and models are available at https://github.com/cmusatyalab/grw-smoothing.