DealMaTe: Multi-Dimensional Material Transfer via Diffusion Transformer

TL;DR

DealMaTe introduces a simplified diffusion framework for multi-dimensional material transfer that eliminates text guidance, reduces computational costs, and achieves high-fidelity results across diverse objects and lighting conditions.

Contribution

Proposes DealMaTe, a novel diffusion-based method using depth, normal, and lighting images for material transfer without text or reference networks.

Findings

Achieves high-fidelity material transfer across various objects and lighting conditions.

Reduces inference latency and computational costs with optimized attention mechanisms.

Demonstrates stable and harmonious results without modifying base model weights.

Abstract

Recently, diffusion-based material transfer methods rely on image fine-tuning or complex architectures with auxiliary networks but face challenges such as text dependency, additional computational costs, and feature misalignment. To address these limitations, we propose \textbf{DealMaTe}, using \underline{\textbf{de}}pth, norm\underline{\textbf{a}}l, and \underline{\textbf{l}}ighting images for \underline{\textbf{ma}}terial \underline{\textbf{t}}ransf\underline{\textbf{e}}r. DealMaTe is a simplified diffusion framework that eliminates text guidance and reference networks. We design a lightweight 3D information injection method, Multi-Dim 3D Shader LoRA, which, without modifying the base model weights, enables compatible control conditions and achieves harmonious and stable results. Additionally, we optimize the attention mechanism with Shader Causal Mutual Attention and key-value (KV) caching to reduce inference latency caused by multiple conditions, improve computational efficiency, and achieve high-quality material transfer results with low architectural complexity. Extensive experiments covering a wide variety of objects and lighting conditions consistently demonstrate that DealMaTe achieves remarkable high-fidelity material transfer under arbitrary input materials. The code is available at https://github.com/haha-lisa/DealMaTe.