Integrating Audio Narrations to Strengthen Domain Generalization in Multimodal First-Person Action Recognition

TL;DR

This paper introduces a multimodal approach combining motion, audio, and appearance features to improve domain generalization in first-person action recognition, leveraging audio narrations and consistency ratings to enhance robustness across environments.

Contribution

The paper presents a novel multimodal framework that integrates audio narrations and consistency ratings to strengthen domain generalization in first-person activity recognition.

Findings

Achieves state-of-the-art results on ARGO1M dataset

Enhances robustness to domain shifts across environments

Validates the effectiveness of audio-text alignment in recognition

Abstract

First-person activity recognition is rapidly growing due to the widespread use of wearable cameras but faces challenges from domain shifts across different environments, such as varying objects or background scenes. We propose a multimodal framework that improves domain generalization by integrating motion, audio, and appearance features. Key contributions include analyzing the resilience of audio and motion features to domain shifts, using audio narrations for enhanced audio-text alignment, and applying consistency ratings between audio and visual narrations to optimize the impact of audio in recognition during training. Our approach achieves state-of-the-art performance on the ARGO1M dataset, effectively generalizing across unseen scenarios and locations.