Action Recognition Using Temporal Shift Module and Ensemble Learning

TL;DR

This paper introduces a high-performing action recognition method combining Temporal Shift Module and ensemble learning, achieving perfect accuracy on a multi-modal dataset for human action classification.

Contribution

It presents the first-rank solution for multi-modal action recognition using TSM, transfer learning, and ensemble techniques, with a focus on multi-modal data integration.

Findings

Achieved perfect top-1 accuracy on test set

Effective multi-modal ensemble boosting performance

Demonstrated efficiency of TSM in action recognition

Abstract

This paper presents the first-rank solution for the Multi-Modal Action Recognition Challenge, part of the Multi-Modal Visual Pattern Recognition Workshop at the \acl{ICPR} 2024. The competition aimed to recognize human actions using a diverse dataset of 20 action classes, collected from multi-modal sources. The proposed approach is built upon the \acl{TSM}, a technique aimed at efficiently capturing temporal dynamics in video data, incorporating multiple data input types. Our strategy included transfer learning to leverage pre-trained models, followed by meticulous fine-tuning on the challenge's specific dataset to optimize performance for the 20 action classes. We carefully selected a backbone network to balance computational efficiency and recognition accuracy and further refined the model using an ensemble technique that integrates outputs from different modalities. This ensemble approach proved crucial in boosting the overall performance. Our solution achieved a perfect top-1 accuracy on the test set, demonstrating the effectiveness of the proposed approach in recognizing human actions across 20 classes. Our code is available online https://github.com/ffyyytt/TSM-MMVPR.