Monocular Dynamic View Synthesis: A Reality Check

TL;DR

This paper critically evaluates monocular dynamic view synthesis methods, revealing that previous results overestimate performance due to protocol leaks, and introduces new metrics, datasets, and protocols for more accurate assessment.

Contribution

It introduces effective multi-view factors, new evaluation metrics, and a diverse dataset, establishing a more realistic benchmark for monocular dynamic view synthesis.

Findings

State-of-the-art methods drop 1-2 dB in masked PSNR without multi-view cues.

Performance drops 4-5 dB on complex motions under the new protocol.

Existing approaches overestimate capabilities due to protocol leaks.

Abstract

We study the recent progress on dynamic view synthesis (DVS) from monocular video. Though existing approaches have demonstrated impressive results, we show a discrepancy between the practical capture process and the existing experimental protocols, which effectively leaks in multi-view signals during training. We define effective multi-view factors (EMFs) to quantify the amount of multi-view signal present in the input capture sequence based on the relative camera-scene motion. We introduce two new metrics: co-visibility masked image metrics and correspondence accuracy, which overcome the issue in existing protocols. We also propose a new iPhone dataset that includes more diverse real-life deformation sequences. Using our proposed experimental protocol, we show that the state-of-the-art approaches observe a 1-2 dB drop in masked PSNR in the absence of multi-view cues and 4-5 dB drop when modeling complex motion. Code and data can be found at https://hangg7.com/dycheck.