Human Action Recognition in Still Images Using ConViT

TL;DR

This paper introduces a hybrid model combining CNNs and Vision Transformers to improve human action recognition in still images by better capturing relationships between image parts, leading to higher accuracy.

Contribution

It proposes a novel module integrating ViT with CNNs, enhancing the extraction of meaningful image parts and relationships for human action recognition.

Findings

Achieved 95.5% mAP on Stanford40 dataset.

Achieved 91.5% mAP on PASCAL VOC 2012 dataset.

Outperformed several state-of-the-art methods.

Abstract

Understanding the relationship between different parts of an image is crucial in a variety of applications, including object recognition, scene understanding, and image classification. Despite the fact that Convolutional Neural Networks (CNNs) have demonstrated impressive results in classifying and detecting objects, they lack the capability to extract the relationship between different parts of an image, which is a crucial factor in Human Action Recognition (HAR). To address this problem, this paper proposes a new module that functions like a convolutional layer that uses Vision Transformer (ViT). In the proposed model, the Vision Transformer can complement a convolutional neural network in a variety of tasks by helping it to effectively extract the relationship among various parts of an image. It is shown that the proposed model, compared to a simple CNN, can extract meaningful parts of an image and suppress the misleading parts. The proposed model has been evaluated on the Stanford40 and PASCAL VOC 2012 action datasets and has achieved 95.5% mean Average Precision (mAP) and 91.5% mAP results, respectively, which are promising compared to other state-of-the-art methods.