4Real-Video: Learning Generalizable Photo-Realistic 4D Video Diffusion

TL;DR

This paper introduces 4Real-Video, a novel framework for generating high-quality, consistent 4D videos using a two-stream diffusion transformer architecture with synchronization layers for improved speed and visual fidelity.

Contribution

The paper presents a new two-stream diffusion transformer architecture with synchronization layers for 4D video generation, enhancing speed and consistency over prior methods.

Findings

Higher inference speed compared to previous models.

Improved visual quality as measured by FVD, CLIP, and VideoScore.

Enhanced temporal and viewpoint consistency demonstrated by evaluation metrics.

Abstract

We propose 4Real-Video, a novel framework for generating 4D videos, organized as a grid of video frames with both time and viewpoint axes. In this grid, each row contains frames sharing the same timestep, while each column contains frames from the same viewpoint. We propose a novel two-stream architecture. One stream performs viewpoint updates on columns, and the other stream performs temporal updates on rows. After each diffusion transformer layer, a synchronization layer exchanges information between the two token streams. We propose two implementations of the synchronization layer, using either hard or soft synchronization. This feedforward architecture improves upon previous work in three ways: higher inference speed, enhanced visual quality (measured by FVD, CLIP, and VideoScore), and improved temporal and viewpoint consistency (measured by VideoScore and Dust3R-Confidence).