Understanding Multimodal Complementarity for Single-Frame Action Anticipation

TL;DR

This paper explores the potential of single-frame visual information for action anticipation, demonstrating that with effective multimodal fusion, it can rival or surpass traditional video-based methods.

Contribution

It introduces AAG+, a refined single-frame anticipation framework that leverages multimodal data and fusion strategies, challenging the need for dense temporal information in action prediction.

Findings

AAG+ outperforms the original AAG in anticipation tasks.

Single-frame methods can match or exceed video-based approaches.

Effective multimodal fusion enhances single-frame action anticipation.

Abstract

Human action anticipation is commonly treated as a video understanding problem, implicitly assuming that dense temporal information is required to reason about future actions. In this work, we challenge this assumption by investigating what can be achieved when action anticipation is constrained to a single visual observation. We ask a fundamental question: how much information about the future is already encoded in a single frame, and how can it be effectively exploited? Building on our prior work on Action Anticipation at a Glimpse (AAG), we conduct a systematic investigation of single-frame action anticipation enriched with complementary sources of information. We analyze the contribution of RGB appearance, depth-based geometric cues, and semantic representations of past actions, and investigate how different multimodal fusion strategies, keyframe selection policies and past-action history sources influence anticipation performance. Guided by these findings, we consolidate the most effective design choices into AAG+, a refined single-frame anticipation framework. Despite operating on a single frame, AAG+ consistently improves upon the original AAG and achieves performance comparable to, or exceeding, that of state-of-the-art video-based methods on challenging anticipation benchmarks including IKEA-ASM, Meccano and Assembly101. Our results offer new insights into the limits and potential of single-frame action anticipation, and clarify when dense temporal modeling is necessary and when a carefully selected glimpse is sufficient.