Cross-View Exocentric to Egocentric Video Synthesis

TL;DR

This paper introduces STA-GAN, a novel bi-directional attention-based generative model that synthesizes egocentric videos from exocentric views by capturing spatial and temporal features, outperforming existing methods.

Contribution

The paper proposes a new STA-GAN model with dual discriminators and attention fusion for cross-view video synthesis, addressing the challenge of view transformation.

Findings

STA-GAN significantly outperforms existing methods on Side2Ego and Top2Ego datasets.

The bi-directional attention fusion improves the quality of generated egocentric videos.

Dual discriminators enhance the robustness of network training.

Abstract

Cross-view video synthesis task seeks to generate video sequences of one view from another dramatically different view. In this paper, we investigate the exocentric (third-person) view to egocentric (first-person) view video generation task. This is challenging because egocentric view sometimes is remarkably different from the exocentric view. Thus, transforming the appearances across the two different views is a non-trivial task. Particularly, we propose a novel Bi-directional Spatial Temporal Attention Fusion Generative Adversarial Network (STA-GAN) to learn both spatial and temporal information to generate egocentric video sequences from the exocentric view. The proposed STA-GAN consists of three parts: temporal branch, spatial branch, and attention fusion. First, the temporal and spatial branches generate a sequence of fake frames and their corresponding features. The fake frames are generated in both downstream and upstream directions for both temporal and spatial branches. Next, the generated four different fake frames and their corresponding features (spatial and temporal branches in two directions) are fed into a novel multi-generation attention fusion module to produce the final video sequence. Meanwhile, we also propose a novel temporal and spatial dual-discriminator for more robust network optimization. Extensive experiments on the Side2Ego and Top2Ego datasets show that the proposed STA-GAN significantly outperforms the existing methods.