RegionRoute: Regional Style Transfer with Diffusion Model

TL;DR

This paper introduces RegionRoute, a diffusion-based style transfer method that achieves precise, localized style application without masks by using attention supervision and novel loss functions, outperforming existing approaches.

Contribution

The paper proposes an attention-supervised diffusion framework with specialized losses and a modular design for efficient, mask-free localized style transfer.

Findings

Achieves accurate, mask-free localized style transfer.

Outperforms existing diffusion-based editing methods.

Introduces Regional Style Editing Score for evaluation.

Abstract

Precise spatial control in diffusion-based style transfer remains challenging. This challenge arises because diffusion models treat style as a global feature and lack explicit spatial grounding of style representations, making it difficult to restrict style application to specific objects or regions. To our knowledge, existing diffusion models are unable to perform true localized style transfer, typically relying on handcrafted masks or multi-stage post-processing that introduce boundary artifacts and limit generalization. To address this, we propose an attention-supervised diffusion framework that explicitly teaches the model where to apply a given style by aligning the attention scores of style tokens with object masks during training. Two complementary objectives, a Focus loss based on KL divergence and a Cover loss using binary cross-entropy, jointly encourage accurate localization and dense coverage. A modular LoRA-MoE design further enables efficient and scalable multi-style adaptation. To evaluate localized stylization, we introduce the Regional Style Editing Score, which measures Regional Style Matching through CLIP-based similarity within the target region and Identity Preservation via masked LPIPS and pixel-level consistency on unedited areas. Experiments show that our method achieves mask-free, single-object style transfer at inference, producing regionally accurate and visually coherent results that outperform existing diffusion-based editing approaches.