ZARRIO @ Ego4D Short Term Object Interaction Anticipation Challenge: Leveraging Affordances and Attention-based models for STA

TL;DR

This paper introduces STAformer, an attention-based model for short-term object interaction anticipation in egocentric videos, incorporating affordance modeling and attention mechanisms to improve prediction accuracy.

Contribution

The paper presents a novel attention-based architecture with modules for modeling affordances and hotspots, advancing the state-of-the-art in egocentric interaction anticipation.

Findings

Achieved 33.5 N mAP on test set

Improved prediction confidence through hotspot modeling

Enhanced performance with multi-scale feature fusion

Abstract

Short-Term object-interaction Anticipation (STA) consists of detecting the location of the next-active objects, the noun and verb categories of the interaction, and the time to contact from the observation of egocentric video. We propose STAformer, a novel attention-based architecture integrating frame-guided temporal pooling, dual image-video attention, and multi-scale feature fusion to support STA predictions from an image-input video pair. Moreover, we introduce two novel modules to ground STA predictions on human behavior by modeling affordances. First, we integrate an environment affordance model which acts as a persistent memory of interactions that can take place in a given physical scene. Second, we predict interaction hotspots from the observation of hands and object trajectories, increasing confidence in STA predictions localized around the hotspot. On the test set, our results obtain a final 33.5 N mAP, 17.25 N+V mAP, 11.77 N+{\delta} mAP and 6.75 Overall top-5 mAP metric when trained on the v2 training dataset.