How Far are AI-generated Videos from Simulating the 3D Visual World: A Learned 3D Evaluation Approach

TL;DR

This paper introduces L3DE, a novel learned 3D evaluation method that objectively measures how well AI-generated videos simulate real 3D visual properties, without needing manual annotations or 3D reconstruction.

Contribution

L3DE is a new, interpretable, and annotation-free approach that assesses 3D visual coherence in AI-generated videos using a trained 3D convolutional network.

Findings

L3DE aligns well with human judgments of video quality.

Persistent gaps exist between real and AI-generated videos in 3D simulation.

L3DE can be used for benchmarking, deepfake detection, and improving video synthesis.

Abstract

Recent advancements in video diffusion models enable the generation of photorealistic videos with impressive 3D consistency and temporal coherence. However, the extent to which these AI-generated videos simulate the 3D visual world remains underexplored. In this paper, we introduce Learned 3D Evaluation (L3DE), an objective, quantifiable, and interpretable method for assessing AI-generated videos' ability to simulate the real world in terms of 3D visual qualities and consistencies, without requiring manually labeled defects or quality annotations. Instead of relying on 3D reconstruction, which is prone to failure with in-the-wild videos, L3DE employs a 3D convolutional network, trained on monocular 3D cues of motion, depth, and appearance, to distinguish real from synthetic videos. Confidence scores from L3DE quantify the gap between real and synthetic videos in terms of 3D visual coherence, while a gradient-based visualization pinpoints unrealistic regions, improving interpretability. We validate L3DE through extensive experiments, demonstrating strong alignment with 3D reconstruction quality and human judgments. Our evaluations on leading generative models (e.g., Kling, Sora, and MiniMax) reveal persistent simulation gaps and subtle inconsistencies. Beyond generative video assessment, L3DE extends to broader applications: benchmarking video generation models, serving as a deepfake detector, and enhancing video synthesis by inpainting flagged inconsistencies. Project page: https://justin-crchang.github.io/l3de-project-page/