Put Myself in Your Shoes: Lifting the Egocentric Perspective from Exocentric Videos

TL;DR

This paper introduces Exo2Ego, a novel framework for translating third-person videos into first-person views, combining structure transformation and diffusion-based hallucination, and provides a new benchmark for this task.

Contribution

The paper proposes a new generative framework for exocentric-to-egocentric video translation and curates a comprehensive benchmark dataset for future research.

Findings

Exo2Ego produces photorealistic egocentric videos with detailed hand manipulation.

It outperforms existing methods in synthesis quality.

The approach generalizes well to new actions.

Abstract

We investigate exocentric-to-egocentric cross-view translation, which aims to generate a first-person (egocentric) view of an actor based on a video recording that captures the actor from a third-person (exocentric) perspective. To this end, we propose a generative framework called Exo2Ego that decouples the translation process into two stages: high-level structure transformation, which explicitly encourages cross-view correspondence between exocentric and egocentric views, and a diffusion-based pixel-level hallucination, which incorporates a hand layout prior to enhance the fidelity of the generated egocentric view. To pave the way for future advancements in this field, we curate a comprehensive exo-to-ego cross-view translation benchmark. It consists of a diverse collection of synchronized ego-exo tabletop activity video pairs sourced from three public datasets: H2O, Aria Pilot, and Assembly101. The experimental results validate that Exo2Ego delivers photorealistic video results with clear hand manipulation details and outperforms several baselines in terms of both synthesis quality and generalization ability to new actions.