Zero-Shot Temporal Interaction Localization for Egocentric Videos

TL;DR

EgoLoc is a novel zero-shot approach for accurately localizing human-object interaction timings in egocentric videos, leveraging adaptive sampling and feedback refinement to outperform existing methods.

Contribution

The paper introduces EgoLoc, a zero-shot TIL method that uses adaptive visual prompts and dynamic feedback for improved accuracy and efficiency in egocentric videos.

Findings

EgoLoc outperforms state-of-the-art baselines in localization accuracy.

The method effectively integrates 2D and 3D cues for initial guesses.

Closed-loop feedback enhances localization precision.

Abstract

Locating human-object interaction (HOI) actions within video serves as the foundation for multiple downstream tasks, such as human behavior analysis and human-robot skill transfer. Current temporal action localization methods typically rely on annotated action and object categories of interactions for optimization, which leads to domain bias and low deployment efficiency. Although some recent works have achieved zero-shot temporal action localization (ZS-TAL) with large vision-language models (VLMs), their coarse-grained estimations and open-loop pipelines hinder further performance improvements for temporal interaction localization (TIL). To address these issues, we propose a novel zero-shot TIL approach dubbed EgoLoc to locate the timings of grasp actions for human-object interaction in egocentric videos. EgoLoc introduces a self-adaptive sampling strategy to generate reasonable visual prompts for VLM reasoning. By absorbing both 2D and 3D observations, it directly samples high-quality initial guesses around the possible contact/separation timestamps of HOI according to 3D hand velocities, leading to high inference accuracy and efficiency. In addition, EgoLoc generates closed-loop feedback from visual and dynamic cues to further refine the localization results. Comprehensive experiments on the publicly available dataset and our newly proposed benchmark demonstrate that EgoLoc achieves better temporal interaction localization for egocentric videos compared to state-of-the-art baselines. We have released our code and relevant data as open-source at https://github.com/IRMVLab/EgoLoc.