RelationAdapter: Learning and Transferring Visual Relation with Diffusion Transformers

TL;DR

This paper introduces RelationAdapter, a lightweight module for Diffusion Transformers that learns and transfers visual relations from minimal examples, enhancing image editing capabilities across diverse tasks.

Contribution

The paper proposes RelationAdapter, a novel module enabling diffusion models to effectively transfer visual relations using minimal examples, and introduces the Relation252K dataset for evaluation.

Findings

RelationAdapter improves editing performance significantly.

The model generalizes well across diverse editing tasks.

The dataset enables comprehensive evaluation of visual relation transfer.

Abstract

Inspired by the in-context learning mechanism of large language models (LLMs), a new paradigm of generalizable visual prompt-based image editing is emerging. Existing single-reference methods typically focus on style or appearance adjustments and struggle with non-rigid transformations. To address these limitations, we propose leveraging source-target image pairs to extract and transfer content-aware editing intent to novel query images. To this end, we introduce RelationAdapter, a lightweight module that enables Diffusion Transformer (DiT) based models to effectively capture and apply visual transformations from minimal examples. We also introduce Relation252K, a comprehensive dataset comprising 218 diverse editing tasks, to evaluate model generalization and adaptability in visual prompt-driven scenarios. Experiments on Relation252K show that RelationAdapter significantly improves the model's ability to understand and transfer editing intent, leading to notable gains in generation quality and overall editing performance.