How You Move Tells What You'll Do: Trajectory-Conditioned Egocentric Prediction

TL;DR

This paper introduces TrajPilot, a model that predicts future egocentric camera trajectories to improve action and goal prediction, outperforming existing baselines across multiple datasets and tasks.

Contribution

The paper demonstrates that using predicted future trajectories as a conditioning signal significantly enhances egocentric action prediction and planning, surpassing language-based and other models.

Findings

TrajPilot outperforms VLM and structured-planner baselines on multiple datasets.

Trajectory-based conditioning improves prediction especially at longer horizons.

The model maintains performance with RGB-only camera-pose estimation.

Abstract

Predicting how a person's first-person view will evolve (what action will follow, what plan completes a task, whether an in-progress shot will score) is fundamentally under-specified: the same context admits many plausible futures, and a model trained to minimize prediction error is forced to hedge or average across them, getting it wrong either way. Two findings shape our approach. First, the future camera trajectory, the path the head carves through space, lets the model commit to one of those futures: it carries the operator's intent in a form fine enough to determine how an action will unfold, substantially outperforming language as a conditioning signal. Second, this same intent makes the trajectory itself partially predictable from the context at hand, enough that trajectory need not be observed at test time to recover most of the gain. We instantiate these findings as TrajPilot, a model that predicts candidate future trajectories from egocentric context and uses them to pilot action prediction in an action-aligned embedding space where language shapes the structure but is never used as a conditioning input. TrajPilot beats VLM and structured-planner baselines on procedural planning across Ego-Exo4D atomic, Ego-Exo4D Keystep, Ego4D GoalStep, and EgoPER, with the trajectory advantage widening with horizon (exactly where prior planners collapse) and holding under RGB-only camera-pose estimation. With the goal masked at inference, the same model performs goal-free anticipation, beating VLM baselines on Ego-Exo4D atomic and extending to EPIC-Kitchens-100 and basketball shot-outcome prediction.