Effects of Different Attention Mechanisms Applied on 3D Models in Video Classification

TL;DR

This study explores how various attention mechanisms affect the performance of 3D CNN models in video classification, emphasizing the importance of temporal features and high-resolution frames.

Contribution

It introduces new attention-augmented variants of 3D CNN models and evaluates their impact on action recognition accuracy using the UCF101 dataset.

Findings

Multi-headed attention improved accuracy to 88.98% on UCF101.

Attention mechanisms influence class-level accuracy differently.

Reducing temporal data affects model performance significantly.

Abstract

Human action recognition has become an important research focus in computer vision due to the wide range of applications where it is used. 3D Resnet-based CNN models, particularly MC3, R3D, and R(2+1)D, have different convolutional filters to extract spatiotemporal features. This paper investigates the impact of reducing the captured knowledge from temporal data, while increasing the resolution of the frames. To establish this experiment, we created similar designs to the three originals, but with a dropout layer added before the final classifier. Secondly, we then developed ten new versions for each one of these three designs. The variants include special attention blocks within their architecture, such as convolutional block attention module (CBAM), temporal convolution networks (TCN), in addition to multi-headed and channel attention mechanisms. The purpose behind that is to observe the extent of the influence each of these blocks has on performance for the restricted-temporal models. The results of testing all the models on UCF101 have shown accuracy of 88.98% for the variant with multiheaded attention added to the modified R(2+1)D. This paper concludes the significance of missing temporal features in the performance of the newly created increased resolution models. The variants had different behavior on class-level accuracy, despite the similarity of their enhancements to the overall performance.