EggHand: A Multimodal Foundation Model for Egocentric Hand Pose Forecasting

TL;DR

EggHand is a multimodal foundation model that predicts future egocentric hand poses by integrating semantic reasoning and dynamic motion modeling, advancing accuracy and robustness in complex scenarios.

Contribution

It introduces a novel framework combining a Vision-Language-Action model with a video-text encoder for improved egocentric hand pose forecasting.

Findings

Sets new state-of-the-art accuracy on EgoExo4D dataset.

Remains robust under severe ego-motion conditions.

Enables controllable prediction using language prompts.

Abstract

Forecasting future 3D hand pose sequences from egocentric video is essential for understanding human intention and enabling embodied applications such as AR/VR assistance and human-robot interaction. However, this task remains a highly challenging problem because egocentric hand motion is driven by complex human intent, exhibits highly dexterous articulations, and is observed under drastic viewpoint shifts induced by ego-motion. In this work, we introduce EggHand, a foundation-model-based framework for egocentric hand pose forecasting that unifies multimodal semantic reasoning with dynamic motion modeling. Our approach couples an action decoder from a Vision-Language-Action (VLA) model, which captures the structured temporal dynamics of hand motion, with an egocentric video-text encoder that provides viewpoint-aware contextual information learned from large-scale first-person video. Together, these components overcome the brittleness of generic visual encoders under ego-motion and enable joint reasoning over motion, context, and high-level intent-without relying on body pose or external tracking. Experiments on the EgoExo4D dataset show that EggHand sets a new state of the art in forecasting accuracy, remains robust under severe ego-motion, and further enables controllable prediction via language-based task prompts. Project page: https://jyoun9.github.io/EggHand