IDArb: Intrinsic Decomposition for Arbitrary Number of Input Views and Illuminations

TL;DR

IDArb is a diffusion-based model that performs intrinsic decomposition on multiple views and illuminations, accurately estimating surface normals and material properties while maintaining multi-view consistency.

Contribution

We introduce IDArb, a novel diffusion model with a cross-view attention module and view-adaptive training for multi-view intrinsic decomposition under varying lighting.

Findings

Outperforms state-of-the-art methods in accuracy and consistency.

Supports diverse downstream tasks like relighting and 3D reconstruction.

Introduces ARB-Objaverse dataset for training and evaluation.

Abstract

Capturing geometric and material information from images remains a fundamental challenge in computer vision and graphics. Traditional optimization-based methods often require hours of computational time to reconstruct geometry, material properties, and environmental lighting from dense multi-view inputs, while still struggling with inherent ambiguities between lighting and material. On the other hand, learning-based approaches leverage rich material priors from existing 3D object datasets but face challenges with maintaining multi-view consistency. In this paper, we introduce IDArb, a diffusion-based model designed to perform intrinsic decomposition on an arbitrary number of images under varying illuminations. Our method achieves accurate and multi-view consistent estimation on surface normals and material properties. This is made possible through a novel cross-view, cross-domain attention module and an illumination-augmented, view-adaptive training strategy. Additionally, we introduce ARB-Objaverse, a new dataset that provides large-scale multi-view intrinsic data and renderings under diverse lighting conditions, supporting robust training. Extensive experiments demonstrate that IDArb outperforms state-of-the-art methods both qualitatively and quantitatively. Moreover, our approach facilitates a range of downstream tasks, including single-image relighting, photometric stereo, and 3D reconstruction, highlighting its broad applications in realistic 3D content creation.