MVInverse: Feed-forward Multi-view Inverse Rendering in Seconds

TL;DR

MVInverse introduces a fast, feed-forward multi-view inverse rendering approach that predicts scene properties from RGB sequences, ensuring coherence across views and improving real-world generalization through a novel finetuning strategy.

Contribution

The paper presents a novel feed-forward framework for multi-view inverse rendering that captures inter-view relationships and introduces a consistency-based finetuning method for real-world scene adaptation.

Findings

Achieves state-of-the-art multi-view consistency and material estimation.

Demonstrates strong generalization to real-world scenes.

Operates efficiently with a single forward pass.

Abstract

Multi-view inverse rendering aims to recover geometry, materials, and illumination consistently across multiple viewpoints. When applied to multi-view images, existing single-view approaches often ignore cross-view relationships, leading to inconsistent results. In contrast, multi-view optimization methods rely on slow differentiable rendering and per-scene refinement, making them computationally expensive and hard to scale. To address these limitations, we introduce a feed-forward multi-view inverse rendering framework that directly predicts spatially varying albedo, metallic, roughness, diffuse shading, and surface normals from sequences of RGB images. By alternating attention across views, our model captures both intra-view long-range lighting interactions and inter-view material consistency, enabling coherent scene-level reasoning within a single forward pass. Due to the scarcity of real-world training data, models trained on existing synthetic datasets often struggle to generalize to real-world scenes. To overcome this limitation, we propose a consistency-based finetuning strategy that leverages unlabeled real-world videos to enhance both multi-view coherence and robustness under in-the-wild conditions. Extensive experiments on benchmark datasets demonstrate that our method achieves state-of-the-art performance in terms of multi-view consistency, material and normal estimation quality, and generalization to real-world imagery. Project page: https://maddog241.github.io/mvinverse-page/