ConViViT -- A Deep Neural Network Combining Convolutions and Factorized Self-Attention for Human Activity Recognition

TL;DR

This paper introduces a hybrid neural network architecture that combines CNNs and Transformers to improve human activity recognition in videos, achieving state-of-the-art accuracy on multiple datasets.

Contribution

The novel hybrid architecture effectively integrates CNNs and Transformers, enhancing video representation and surpassing previous methods in activity recognition accuracy.

Findings

Achieved 90.05% on HMDB51 dataset.

Achieved 99.6% on UCF101 dataset.

Achieved 95.09% on ETRI-Activity3D dataset.

Abstract

The Transformer architecture has gained significant popularity in computer vision tasks due to its capacity to generalize and capture long-range dependencies. This characteristic makes it well-suited for generating spatiotemporal tokens from videos. On the other hand, convolutions serve as the fundamental backbone for processing images and videos, as they efficiently aggregate information within small local neighborhoods to create spatial tokens that describe the spatial dimension of a video. While both CNN-based architectures and pure transformer architectures are extensively studied and utilized by researchers, the effective combination of these two backbones has not received comparable attention in the field of activity recognition. In this research, we propose a novel approach that leverages the strengths of both CNNs and Transformers in an hybrid architecture for performing activity recognition using RGB videos. Specifically, we suggest employing a CNN network to enhance the video representation by generating a 128-channel video that effectively separates the human performing the activity from the background. Subsequently, the output of the CNN module is fed into a transformer to extract spatiotemporal tokens, which are then used for classification purposes. Our architecture has achieved new SOTA results with 90.05 \%, 99.6\%, and 95.09\% on HMDB51, UCF101, and ETRI-Activity3D respectively.