Continual Multimodal Egocentric Activity Recognition via Modality-Aware Novel Detection

TL;DR

This paper introduces MAND, a novel framework for open-world continual learning in egocentric activity recognition that effectively detects novel activities by leveraging modality-aware scoring and stabilizing modality-specific features.

Contribution

The paper proposes MAND, a modality-aware framework that improves novelty detection and knowledge retention in multimodal egocentric continual learning.

Findings

MAND improves novel activity detection AUC by up to 10%.

MAND increases known-class classification accuracy by up to 2.8%.

The approach effectively exploits multimodal cues and mitigates catastrophic forgetting.

Abstract

Multimodal egocentric activity recognition integrates visual and inertial cues for robust first-person behavior understanding. However, deploying such systems in open-world environments requires detecting novel activities while continuously learning from non-stationary streams. Existing methods rely on the main logits for novelty scoring, without fully exploiting the complementary evidence available from individual modalities. Because these logits are often dominated by RGB, cues from other modalities, particularly IMU, remain underutilized, and this imbalance worsens over time under catastrophic forgetting. To address this, we propose MAND, a modality-aware framework for multimodal egocentric open-world continual learning. At inference, Modality-aware Adaptive Scoring (MoAS) estimates sample-wise modality reliability from energy scores and adaptively integrates modality logits to better exploit complementary modality cues for novelty detection. During training, Modality-wise Representation Stabilization Training (MoRST) preserves modality-specific discriminability across tasks via auxiliary heads and modality-wise logit distillation. Experiments on a public multimodal egocentric benchmark show that MAND improves novel activity detection AUC by up to 10\% and known-class classification accuracy by up to 2.8\% over state-of-the-art baselines.