Multimodal Cross-Domain Few-Shot Learning for Egocentric Action Recognition

TL;DR

This paper introduces MM-CDFSL, a multimodal, domain-adaptive, and computationally efficient method for egocentric action recognition in cross-domain few-shot learning, leveraging multimodal distillation and ensemble masked inference to improve accuracy and speed.

Contribution

The paper proposes a novel multimodal distillation framework with ensemble masked inference for cross-domain few-shot egocentric action recognition, addressing domain gap and computational efficiency.

Findings

Outperforms state-of-the-art CD-FSL methods by over 6 points in accuracy.

Achieves 2.2 times faster inference speed.

Effectively adapts to unlabeled target domain data.

Abstract

We address a novel cross-domain few-shot learning task (CD-FSL) with multimodal input and unlabeled target data for egocentric action recognition. This paper simultaneously tackles two critical challenges associated with egocentric action recognition in CD-FSL settings: (1) the extreme domain gap in egocentric videos (e.g., daily life vs. industrial domain) and (2) the computational cost for real-world applications. We propose MM-CDFSL, a domain-adaptive and computationally efficient approach designed to enhance adaptability to the target domain and improve inference cost. To address the first challenge, we propose the incorporation of multimodal distillation into the student RGB model using teacher models. Each teacher model is trained independently on source and target data for its respective modality. Leveraging only unlabeled target data during multimodal distillation enhances the student model's adaptability to the target domain. We further introduce ensemble masked inference, a technique that reduces the number of input tokens through masking. In this approach, ensemble prediction mitigates the performance degradation caused by masking, effectively addressing the second issue. Our approach outperformed the state-of-the-art CD-FSL approaches with a substantial margin on multiple egocentric datasets, improving by an average of 6.12/6.10 points for 1-shot/5-shot settings while achieving $2.2$ times faster inference speed. Project page: https://masashi-hatano.github.io/MM-CDFSL/