Harmonizing Attention: Training-free Texture-aware Geometry Transfer

TL;DR

This paper presents Harmonizing Attention, a training-free method using diffusion models with modified self-attention layers to transfer geometry features across materials while preserving textures, without fine-tuning.

Contribution

It introduces a novel dual-attention mechanism that enables geometry transfer independent of surface textures without requiring model training or fine-tuning.

Findings

Effective geometry transfer across materials demonstrated.

Maintains material-specific textures during transfer.

Operates without additional training or fine-tuning.

Abstract

Extracting geometry features from photographic images independently of surface texture and transferring them onto different materials remains a complex challenge. In this study, we introduce Harmonizing Attention, a novel training-free approach that leverages diffusion models for texture-aware geometry transfer. Our method employs a simple yet effective modification of self-attention layers, allowing the model to query information from multiple reference images within these layers. This mechanism is seamlessly integrated into the inversion process as Texture-aligning Attention and into the generation process as Geometry-aligning Attention. This dual-attention approach ensures the effective capture and transfer of material-independent geometry features while maintaining material-specific textural continuity, all without the need for model fine-tuning.