Spatially Prompted Visual Trajectory Prediction for Egocentric Manipulation

TL;DR

This paper introduces a new spatial prompting framework for egocentric manipulation, utilizing initial spatial cues to predict future trajectories, supported by a novel dataset and a specialized model.

Contribution

It formalizes the SP-VTP problem, creates the EgoSPT dataset, and proposes SPOT, a model that improves trajectory prediction with spatial prompts in egocentric scenes.

Findings

SPOT outperforms non-prompted baselines in cross-scene trajectory prediction.

EgoSPT dataset provides annotated egocentric manipulation trajectories with spatial prompts.

Spatial prompts enhance the generalization of trajectory prediction models.

Abstract

Robotic manipulation is often specified through language instructions or task identifiers, yet cluttered environments with similar objects are better handled by spatially indicating what to move and where to place it. Addressing the vision-centric challenge of object and goal specification, we present, to the best of our knowledge, the first formalization of Spatially Prompted Visual Trajectory Prediction (SP-VTP). This novel setting utilizes initial spatial prompts (like bounding boxes or points) to define task objectives, tasking the model with forecasting future end-effector trajectories from egocentric streams. To study this problem, we collect and annotate EgoSPT, a dataset of egocentric spatially prompted manipulation trajectories with first-frame object and target grounding annotations and recovered 3D end-effector motion. SP-VTP is challenging because the task specification is static, while the scene configuration evolves over time. To solve this problem, we propose SPOT(Spatially Prompted Object-Target Policy), which combines a task encoder for first-frame visual and coordinate spatial prompts, an observation encoder for current visual and history context, and a trajectory generator for future end-effector motion. Experiments under strict scene-level splits show that SPOT improves cross-scene trajectory prediction over non-prompted or single-source prompted baselines. Together, EgoSPT and SPOT establish a new spatial prompting problem SP-VTP, as a simple and scalable task condition for egocentric manipulation.