MatSwap: Light-aware material transfers in images

TL;DR

MatSwap is a novel light-aware diffusion-based method that enables photorealistic transfer of materials onto specific surfaces in images without requiring manual annotations or explicit UV mapping, leveraging a fine-tuned large-scale model.

Contribution

We introduce a light- and geometry-aware diffusion model for material transfer that learns from synthetic data, enabling realistic editing without manual annotations or scene geometry.

Findings

Outperforms recent methods qualitatively and quantitatively

Works effectively on both synthetic and real images

Preserves scene identity during material transfer

Abstract

We present MatSwap, a method to transfer materials to designated surfaces in an image photorealistically. Such a task is non-trivial due to the large entanglement of material appearance, geometry, and lighting in a photograph. In the literature, material editing methods typically rely on either cumbersome text engineering or extensive manual annotations requiring artist knowledge and 3D scene properties that are impractical to obtain. In contrast, we propose to directly learn the relationship between the input material -- as observed on a flat surface -- and its appearance within the scene, without the need for explicit UV mapping. To achieve this, we rely on a custom light- and geometry-aware diffusion model. We fine-tune a large-scale pre-trained text-to-image model for material transfer using our synthetic dataset, preserving its strong priors to ensure effective generalization to real images. As a result, our method seamlessly integrates a desired material into the target location in the photograph while retaining the identity of the scene. We evaluate our method on synthetic and real images and show that it compares favorably to recent work both qualitatively and quantitatively. We release our code and data on https://github.com/astra-vision/MatSwap