MU-MAE: Multimodal Masked Autoencoders-Based One-Shot Learning

TL;DR

Mu-MAE introduces a self-supervised multimodal autoencoder with a novel masking strategy, enabling effective one-shot human activity recognition from video and sensor data without external datasets.

Contribution

The paper proposes Mu-MAE, a multimodal masked autoencoder with synchronized masking for self-supervised pretraining and a cross-attention fusion layer for improved one-shot classification.

Findings

Achieves up to 80.17% accuracy on MMAct one-shot classification

Outperforms existing approaches without external data

Effective spatiotemporal feature learning through novel masking strategy

Abstract

With the exponential growth of multimedia data, leveraging multimodal sensors presents a promising approach for improving accuracy in human activity recognition. Nevertheless, accurately identifying these activities using both video data and wearable sensor data presents challenges due to the labor-intensive data annotation, and reliance on external pretrained models or additional data. To address these challenges, we introduce Multimodal Masked Autoencoders-Based One-Shot Learning (Mu-MAE). Mu-MAE integrates a multimodal masked autoencoder with a synchronized masking strategy tailored for wearable sensors. This masking strategy compels the networks to capture more meaningful spatiotemporal features, which enables effective self-supervised pretraining without the need for external data. Furthermore, Mu-MAE leverages the representation extracted from multimodal masked autoencoders as prior information input to a cross-attention multimodal fusion layer. This fusion layer emphasizes spatiotemporal features requiring attention across different modalities while highlighting differences from other classes, aiding in the classification of various classes in metric-based one-shot learning. Comprehensive evaluations on MMAct one-shot classification show that Mu-MAE outperforms all the evaluated approaches, achieving up to an 80.17% accuracy for five-way one-shot multimodal classification, without the use of additional data.