View while Moving: Efficient Video Recognition in Long-untrimmed Videos

TL;DR

This paper introduces a novel 'View while Moving' paradigm for efficient long-untrimmed video recognition, reducing raw frame access to once and unifying coarse and fine-grained analysis for improved accuracy and efficiency.

Contribution

It proposes a new recognition paradigm inspired by human cognition that unifies sampling and recognition, enabling more efficient long-video analysis with hierarchical semantic modeling.

Findings

Outperforms state-of-the-art methods in accuracy and efficiency

Reduces raw frame access to a single pass during inference

Demonstrates effective hierarchical semantic understanding

Abstract

Recent adaptive methods for efficient video recognition mostly follow the two-stage paradigm of "preview-then-recognition" and have achieved great success on multiple video benchmarks. However, this two-stage paradigm involves two visits of raw frames from coarse-grained to fine-grained during inference (cannot be parallelized), and the captured spatiotemporal features cannot be reused in the second stage (due to varying granularity), being not friendly to efficiency and computation optimization. To this end, inspired by human cognition, we propose a novel recognition paradigm of "View while Moving" for efficient long-untrimmed video recognition. In contrast to the two-stage paradigm, our paradigm only needs to access the raw frame once. The two phases of coarse-grained sampling and fine-grained recognition are combined into unified spatiotemporal modeling, showing great performance. Moreover, we investigate the properties of semantic units in video and propose a hierarchical mechanism to efficiently capture and reason about the unit-level and video-level temporal semantics in long-untrimmed videos respectively. Extensive experiments on both long-untrimmed and short-trimmed videos demonstrate that our approach outperforms state-of-the-art methods in terms of accuracy as well as efficiency, yielding new efficiency and accuracy trade-offs for video spatiotemporal modeling.