Refa\c{c}ade: Editing Object with Given Reference Texture

TL;DR

Refacade introduces a novel method for precise and controllable object retexturing in images and videos by removing appearance information and disrupting global layout, outperforming existing techniques.

Contribution

The paper proposes Refacade, a new approach that enhances texture transfer control by removing appearance cues and disrupting global layout, enabling better editing in visual media.

Findings

Outperforms strong baselines in visual quality and control.

Effective in both image and video retexturing tasks.

Demonstrates superior quantitative and human evaluation results.

Abstract

Recent advances in diffusion models have brought remarkable progress in image and video editing, yet some tasks remain underexplored. In this paper, we introduce a new task, Object Retexture, which transfers local textures from a reference object to a target object in images or videos. To perform this task, a straightforward solution is to use ControlNet conditioned on the source structure and the reference texture. However, this approach suffers from limited controllability for two reasons: conditioning on the raw reference image introduces unwanted structural information, and it fails to disentangle the visual texture and structure information of the source. To address this problem, we propose Refa\c{c}ade, a method that consists of two key designs to achieve precise and controllable texture transfer in both images and videos. First, we employ a texture remover trained on paired textured/untextured 3D mesh renderings to remove appearance information while preserving the geometry and motion of source videos. Second, we disrupt the reference global layout using a jigsaw permutation, encouraging the model to focus on local texture statistics rather than the global layout of the object. Extensive experiments demonstrate superior visual quality, precise editing, and controllability, outperforming strong baselines in both quantitative and human evaluations. Code is available at https://github.com/fishZe233/Refacade.