Alchemist: Parametric Control of Material Properties with Diffusion Models

TL;DR

This paper introduces Alchemist, a method leveraging diffusion models and synthetic datasets to enable precise control over material properties in real images, facilitating realistic editing of attributes like roughness and transparency.

Contribution

It presents a novel approach combining synthetic data and diffusion models for controllable material editing in real images, addressing dataset limitations.

Findings

Effective control of material attributes in real images.

Preserves non-material attributes during editing.

Potential application to material edited NeRFs.

Abstract

We propose a method to control material attributes of objects like roughness, metallic, albedo, and transparency in real images. Our method capitalizes on the generative prior of text-to-image models known for photorealism, employing a scalar value and instructions to alter low-level material properties. Addressing the lack of datasets with controlled material attributes, we generated an object-centric synthetic dataset with physically-based materials. Fine-tuning a modified pre-trained text-to-image model on this synthetic dataset enables us to edit material properties in real-world images while preserving all other attributes. We show the potential application of our model to material edited NeRFs.