The Devil is in Attention Sharing: Improving Complex Non-rigid Image Editing Faithfulness via Attention Synergy

TL;DR

SynPS enhances non-rigid image editing with diffusion models by dynamically balancing positional and semantic cues, significantly improving editing faithfulness and avoiding over- or under-editing.

Contribution

We introduce SynPS, a novel attention synergy method that adaptively combines positional embeddings and semantic information for more faithful complex image editing.

Findings

Outperforms existing methods in editing faithfulness.

Effectively balances semantic and positional influences.

Reduces over- and under-editing in non-rigid edits.

Abstract

Training-free image editing with large diffusion models has become practical, yet faithfully performing complex non-rigid edits (e.g., pose or shape changes) remains highly challenging. We identify a key underlying cause: attention collapse in existing attention sharing mechanisms, where either positional embeddings or semantic features dominate visual content retrieval, leading to over-editing or under-editing. To address this issue, we introduce SynPS, a method that Synergistically leverages Positional embeddings and Semantic information for faithful non-rigid image editing. We first propose an editing measurement that quantifies the required editing magnitude at each denoising step. Based on this measurement, we design an attention synergy pipeline that dynamically modulates the influence of positional embeddings, enabling SynPS to balance semantic modifications and fidelity preservation. By adaptively integrating positional and semantic cues, SynPS effectively avoids both over- and under-editing. Extensive experiments on public and newly curated benchmarks demonstrate the superior performance and faithfulness of our approach.