An Effective End-to-End Solution for Multimodal Action Recognition

TL;DR

This paper presents a comprehensive end-to-end multimodal action recognition system that leverages data augmentation, transfer learning, efficient spatial-temporal feature extraction, and prediction ensemble techniques to achieve state-of-the-art accuracy.

Contribution

The proposed solution introduces an integrated approach combining data enhancement, transfer learning, efficient multimodal feature extraction, and ensemble methods for improved action recognition performance.

Findings

Achieved 99% Top-1 accuracy on the leaderboard.

Utilized transfer learning with RGB datasets for better model adaptation.

Combined multiple prediction techniques to enhance accuracy.

Abstract

Recently, multimodal tasks have strongly advanced the field of action recognition with their rich multimodal information. However, due to the scarcity of tri-modal data, research on tri-modal action recognition tasks faces many challenges. To this end, we have proposed a comprehensive multimodal action recognition solution that effectively utilizes multimodal information. First, the existing data are transformed and expanded by optimizing data enhancement techniques to enlarge the training scale. At the same time, more RGB datasets are used to pre-train the backbone network, which is better adapted to the new task by means of transfer learning. Secondly, multimodal spatial features are extracted with the help of 2D CNNs and combined with the Temporal Shift Module (TSM) to achieve multimodal spatial-temporal feature extraction comparable to 3D CNNs and improve the computational efficiency. In addition, common prediction enhancement methods, such as Stochastic Weight Averaging (SWA), Ensemble and Test-Time augmentation (TTA), are used to integrate the knowledge of models from different training periods of the same architecture and different architectures, so as to predict the actions from different perspectives and fully exploit the target information. Ultimately, we achieved the Top-1 accuracy of 99% and the Top-5 accuracy of 100% on the competition leaderboard, demonstrating the superiority of our solution.