From Synchrony to Sequence: Exo-to-Ego Generation via Interpolation

TL;DR

This paper introduces Syn2Seq-Forcing, a sequential modeling approach for exo-to-ego video generation that interpolates between source and target videos to address synchronization-induced discontinuities.

Contribution

It reframes exo-to-ego generation as sequence modeling, enabling diffusion models to better capture coherent transitions and unify cross-view video synthesis tasks.

Findings

Interpolating videos alone improves transition coherence.

The approach effectively handles spatio-temporal discontinuities.

Framework unifies exo-to-ego and ego-to-exo generation.

Abstract

Exo-to-Ego video generation aims to synthesize a first-person video from a synchronized third-person view and corresponding camera poses. While paired supervision is available, synchronized exo-ego data inherently introduces substantial spatio-temporal and geometric discontinuities, violating the smooth-motion assumptions of standard video generation benchmarks. We identify this synchronization-induced jump as the central challenge and propose Syn2Seq-Forcing, a sequential formulation that interpolates between the source and target videos to form a single continuous signal. By reframing Exo2Ego as sequential signal modeling rather than a conventional condition-output task, our approach enables diffusion-based sequence models, e.g. Diffusion Forcing Transformers (DFoT), to capture coherent transitions across frames more effectively. Empirically, we show that interpolating only the videos, without performing pose interpolation already produces significant improvements, emphasizing that the dominant difficulty arises from spatio-temporal discontinuities. Beyond immediate performance gains, this formulation establishes a general and flexible framework capable of unifying both Exo2Ego and Ego2Exo generation within a single continuous sequence model, providing a principled foundation for future research in cross-view video synthesis.