Multimodal Foundation Model for Cross-Modal Retrieval and Activity Recognition Tasks

TL;DR

This paper introduces AURA-MFM, a multimodal foundation model that integrates video, motion capture, IMU, and text data to improve detailed human activity analysis and recognition, especially in zero-shot scenarios.

Contribution

The paper presents a novel multimodal foundation model that combines four data modalities, including third-person video and motion capture, to enhance activity understanding beyond existing models.

Findings

Outperforms existing methods in retrieval and activity recognition tasks.

Achieves a zero-shot action recognition F1-score of 0.6226.

Zero-shot accuracy of 0.7320 significantly higher than previous approaches.

Abstract

In recent years, the widespread adoption of wearable devices has highlighted the growing importance of behavior analysis using IMU. While applications span diverse fields such as healthcare and robotics, recent studies have increasingly focused on multimodal analysis, in addition to unimodal analysis. Several studies have proposed multimodal foundation models that incorporate first-person video and text data; however, these models still fall short in providing a detailed analysis of full-body human activity. To address this limitation, we propose Activity Understanding and Representations Alignment - Multimodal Foundation Model (AURA-MFM), a foundational model integrating four modalities: third-person video, motion capture, IMU, and text. By incorporating third-person video and motion capture data, the model enables a detailed and multidimensional understanding of human activity, which first-person perspectives alone fail to capture. Additionally, a Transformer-based IMU encoder is employed to enhance the model's overall performance. Experimental evaluations on retrieval and activity recognition tasks demonstrate that our model surpasses existing methods. Notably, in the zero-shot classification for action recognition, our method achieved significantly higher performance, with an F1-score of 0.6226 and an accuracy of 0.7320, whereas the existing method recorded an F1-score of 0.0747 and an accuracy of 0.1961.