# Ego-centric Predictive Model Conditioned on Hand Trajectories

## TL;DR

This paper introduces a unified two-stage predictive model for egocentric scenarios that jointly predicts future actions and visual outcomes conditioned on hand trajectories, advancing human activity understanding and robotic manipulation.

## Contribution

The paper presents the first unified framework that explicitly models both action prediction and visual future in egocentric settings, conditioned on hand trajectories.

## Key findings

- Outperforms state-of-the-art in action prediction
- Achieves more plausible future video generation
- Effective on Ego4D, BridgeData, and RLBench datasets

## Abstract

In egocentric scenarios, anticipating both the next action and its visual outcome is essential for understanding human-object interactions and for enabling robotic planning. However, existing paradigms fall short of jointly modeling these aspects. Vision-Language-Action (VLA) models focus on action prediction but lack explicit modeling of how actions influence the visual scene, while video prediction models generate future frames without conditioning on specific actions, often resulting in implausible or contextually inconsistent outcomes. To bridge this gap, we propose a unified two-stage predictive framework that jointly models action and visual future in egocentric scenarios, conditioned on hand trajectories. In the first stage, we perform consecutive state modeling to process heterogeneous inputs (visual observations, language, and action history) and explicitly predict future hand trajectories. In the second stage, we introduce causal cross-attention to fuse multi-modal cues, leveraging inferred action signals to guide an image-based Latent Diffusion Model (LDM) for frame-by-frame future video generation. Our approach is the first unified model designed to handle both egocentric human activity understanding and robotic manipulation tasks, providing explicit predictions of both upcoming actions and their visual consequences. Extensive experiments on Ego4D, BridgeData, and RLBench demonstrate that our method outperforms state-of-the-art baselines in both action prediction and future video synthesis.