Short-term Object Interaction Anticipation with Disentangled Object Detection @ Ego4D Short Term Object Interaction Anticipation Challenge

TL;DR

This paper introduces SOIA-DOD, a method for short-term object interaction anticipation in egocentric videos, which decomposes the task into object detection, classification, and timing prediction, outperforming previous models.

Contribution

The paper proposes a novel approach that combines object detection with transformer-based prediction for interaction anticipation, achieving state-of-the-art results.

Findings

Outperforms existing models on the challenge test set.

Achieves top performance in predicting next active objects and interactions.

Ranks third in overall top-5 mAP including time-to-contact predictions.

Abstract

Short-term object interaction anticipation is an important task in egocentric video analysis, including precise predictions of future interactions and their timings as well as the categories and positions of the involved active objects. To alleviate the complexity of this task, our proposed method, SOIA-DOD, effectively decompose it into 1) detecting active object and 2) classifying interaction and predicting their timing. Our method first detects all potential active objects in the last frame of egocentric video by fine-tuning a pre-trained YOLOv9. Then, we combine these potential active objects as query with transformer encoder, thereby identifying the most promising next active object and predicting its future interaction and time-to-contact. Experimental results demonstrate that our method outperforms state-of-the-art models on the challenge test set, achieving the best performance in predicting next active objects and their interactions. Finally, our proposed ranked the third overall top-5 mAP when including time-to-contact predictions. The source code is available at https://github.com/KeenyJin/SOIA-DOD.