EgoPrompt: Prompt Learning for Egocentric Action Recognition

TL;DR

EgoPrompt introduces a prompt learning framework that models the semantic and contextual relationships between verbs and nouns in egocentric action recognition, achieving state-of-the-art results across multiple datasets.

Contribution

The paper proposes a novel prompt learning approach with a Unified Prompt Pool and Diverse Pool Criteria to improve the integration of component-specific knowledge in egocentric action recognition.

Findings

Achieves state-of-the-art performance on Ego4D, EPIC-Kitchens, and EGTEA datasets.

Effectively models cross-component interactions via attention-based fusion.

Demonstrates strong generalization in cross-dataset and base-to-novel scenarios.

Abstract

Driven by the increasing demand for applications in augmented and virtual reality, egocentric action recognition has emerged as a prominent research area. It is typically divided into two subtasks: recognizing the performed behavior (i.e., verb component) and identifying the objects being acted upon (i.e., noun component) from the first-person perspective. However, most existing approaches treat these two components as independent classification tasks, focusing on extracting component-specific knowledge while overlooking their inherent semantic and contextual relationships, leading to fragmented representations and sub-optimal generalization capability. To address these challenges, we propose a prompt learning-based framework, EgoPrompt, to conduct the egocentric action recognition task. Building on the existing prompting strategy to capture the component-specific knowledge, we construct a Unified Prompt Pool space to establish interaction between the two types of component representations. Specifically, the component representations (from verbs and nouns) are first decomposed into fine-grained patterns with the prompt pair form. Then, these pattern-level representations are fused through an attention-based mechanism to facilitate cross-component interaction. To ensure the prompt pool is informative, we further introduce a novel training objective, Diverse Pool Criteria. This objective realizes our goals from two perspectives: Prompt Selection Frequency Regularization and Prompt Knowledge Orthogonalization. Extensive experiments are conducted on the Ego4D, EPIC-Kitchens, and EGTEA datasets. The results consistently show that EgoPrompt achieves state-of-the-art performance across within-dataset, cross-dataset, and base-to-novel generalization benchmarks.