MVGBench: Comprehensive Benchmark for Multi-view Generation Models

TL;DR

MVGBench introduces a comprehensive evaluation framework for multi-view image generation models, focusing on 3D consistency, robustness, and generalization, revealing limitations of current methods and guiding the development of improved models like ViFiGen.

Contribution

The paper presents MVGBench, a novel benchmark with a 3D self-consistency metric, and introduces ViFiGen, a new model that surpasses existing MVGs in 3D consistency.

Findings

Existing MVGs show limited robustness and generalization.

The 3D self-consistency metric effectively evaluates multi-view models.

ViFiGen outperforms other models in 3D consistency.

Abstract

We propose MVGBench, a comprehensive benchmark for multi-view image generation models (MVGs) that evaluates 3D consistency in geometry and texture, image quality, and semantics (using vision language models). Recently, MVGs have been the main driving force in 3D object creation. However, existing metrics compare generated images against ground truth target views, which is not suitable for generative tasks where multiple solutions exist while differing from ground truth. Furthermore, different MVGs are trained on different view angles, synthetic data and specific lightings -- robustness to these factors and generalization to real data are rarely evaluated thoroughly. Without a rigorous evaluation protocol, it is also unclear what design choices contribute to the progress of MVGs. MVGBench evaluates three different aspects: best setup performance, generalization to real data and robustness. Instead of comparing against ground truth, we introduce a novel 3D self-consistency metric which compares 3D reconstructions from disjoint generated multi-views. We systematically compare 12 existing MVGs on 4 different curated real and synthetic datasets. With our analysis, we identify important limitations of existing methods specially in terms of robustness and generalization, and we find the most critical design choices. Using the discovered best practices, we propose ViFiGen, a method that outperforms all evaluated MVGs on 3D consistency. Our code, model, and benchmark suite will be publicly released.