MatE: Material Extraction from Single-Image via Geometric Prior

TL;DR

MatE is a novel method that generates high-quality, tileable PBR materials from a single real-world image, using geometric priors and diffusion models to produce detailed material maps.

Contribution

We introduce MatE, a new approach that produces complete PBR material maps from a single image with minimal user input, handling real-world conditions and unknown illumination.

Findings

Successfully generates realistic material maps from casual images.

Demonstrates robustness across synthetic and real-world datasets.

Achieves invariance to illumination and perspective variations.

Abstract

The creation of high-fidelity, physically-based rendering (PBR) materials remains a bottleneck in many graphics pipelines, typically requiring specialized equipment and expert-driven post-processing. To democratize this process, we present MatE, a novel method for generating tileable PBR materials from a single image taken under unconstrained, real-world conditions. Given an image and a user-provided mask, MatE first performs coarse rectification using an estimated depth map as a geometric prior, and then employs a dual-branch diffusion model. Leveraging a learned consistency from rotation-aligned and scale-aligned training data, this model further rectify residual distortions from the coarse result and translate it into a complete set of material maps, including albedo, normal, roughness and height. Our framework achieves invariance to the unknown illumination and perspective of the input image, allowing for the recovery of intrinsic material properties from casual captures. Through comprehensive experiments on both synthetic and real-world data, we demonstrate the efficacy and robustness of our approach, enabling users to create realistic materials from real-world image.