Multi-level and Multi-modal Action Anticipation

TL;DR

This paper introduces a multi-modal, hierarchical approach to action anticipation that combines visual and textual cues, improving prediction accuracy by explicitly modeling semantic information and addressing label inaccuracies.

Contribution

It proposes a novel multi-modal and multi-level framework for action anticipation, integrating textual and visual data with hierarchical semantics and a label refinement mechanism.

Findings

Achieved state-of-the-art anticipation accuracy on benchmark datasets.

Improved accuracy by an average of 3.08% over existing methods.

Demonstrated effectiveness of multi-modal and hierarchical modeling in action prediction.

Abstract

Action anticipation, the task of predicting future actions from partially observed videos, is crucial for advancing intelligent systems. Unlike action recognition, which operates on fully observed videos, action anticipation must handle incomplete information. Hence, it requires temporal reasoning, and inherent uncertainty handling. While recent advances have been made, traditional methods often focus solely on visual modalities, neglecting the potential of integrating multiple sources of information. Drawing inspiration from human behavior, we introduce \textit{Multi-level and Multi-modal Action Anticipation (m\&m-Ant)}, a novel multi-modal action anticipation approach that combines both visual and textual cues, while explicitly modeling hierarchical semantic information for more accurate predictions. To address the challenge of inaccurate coarse action labels, we propose a fine-grained label generator paired with a specialized temporal consistency loss function to optimize performance. Extensive experiments on widely used datasets, including Breakfast, 50 Salads, and DARai, demonstrate the effectiveness of our approach, achieving state-of-the-art results with an average anticipation accuracy improvement of 3.08\% over existing methods. This work underscores the potential of multi-modal and hierarchical modeling in advancing action anticipation and establishes a new benchmark for future research in the field. Our code is available at: https://github.com/olivesgatech/mM-ant.