Flatten: Video Action Recognition is an Image Classification task

TL;DR

This paper introduces Flatten, a novel module that transforms 3D video data into 2D representations, enabling the use of image classification models for efficient video action recognition with improved performance.

Contribution

Flatten provides a simple, plug-and-play solution to adapt image understanding models for video recognition by converting spatiotemporal data into 2D, reducing complexity and enhancing accuracy.

Findings

Significant performance improvements on Kinetics-400, Something-Something v2, and HMDB-51 datasets.

Effective integration with models like ResNet, SwinV2, and Uniformer.

Simplifies video recognition by leveraging existing image classification architectures.

Abstract

In recent years, video action recognition, as a fundamental task in the field of video understanding, has been deeply explored by numerous researchers.Most traditional video action recognition methods typically involve converting videos into three-dimensional data that encapsulates both spatial and temporal information, subsequently leveraging prevalent image understanding models to model and analyze these data. However,these methods have significant drawbacks. Firstly, when delving into video action recognition tasks, image understanding models often need to be adapted accordingly in terms of model architecture and preprocessing for these spatiotemporal tasks; Secondly, dealing with high-dimensional data often poses greater challenges and incurs higher time costs compared to its lower-dimensional counterparts.To bridge the gap between image-understanding and video-understanding tasks while simplifying the complexity of video comprehension, we introduce a novel video representation architecture, Flatten, which serves as a plug-and-play module that can be seamlessly integrated into any image-understanding network for efficient and effective 3D temporal data modeling.Specifically, by applying specific flattening operations (e.g., row-major transform), 3D spatiotemporal data is transformed into 2D spatial information, and then ordinary image understanding models are used to capture temporal dynamic and spatial semantic information, which in turn accomplishes effective and efficient video action recognition. Extensive experiments on commonly used datasets (Kinetics-400, Something-Something v2, and HMDB-51) and three classical image classification models (Uniformer, SwinV2, and ResNet), have demonstrated that embedding Flatten provides a significant performance improvements over original model.