CARE-Edit: Condition-Aware Routing of Experts for Contextual Image Editing

TL;DR

CARE-Edit introduces a dynamic, condition-aware expert routing framework for diffusion-based image editing, significantly improving task-specific performance and reducing artifacts by aligning model computation with editing conditions.

Contribution

It proposes a novel latent-attention router that dynamically assigns diffusion tokens to specialized experts based on multi-modal conditions, enhancing contextual image editing capabilities.

Findings

Outperforms existing methods on various editing tasks

Effectively reduces artifacts like color bleeding and style drift

Demonstrates the importance of dynamic expert allocation in multi-condition editing

Abstract

Unified diffusion editors often rely on a fixed, shared backbone for diverse tasks, suffering from task interference and poor adaptation to heterogeneous demands (e.g., local vs global, semantic vs photometric). In particular, prevalent ControlNet and OmniControl variants combine multiple conditioning signals (e.g., text, mask, reference) via static concatenation or additive adapters which cannot dynamically prioritize or suppress conflicting modalities, thus resulting in artifacts like color bleeding across mask boundaries, identity or style drift, and unpredictable behavior under multi-condition inputs. To address this, we propose Condition-Aware Routing of Experts (CARE-Edit) that aligns model computation with specific editing competencies. At its core, a lightweight latent-attention router assigns encoded diffusion tokens to four specialized experts--Text, Mask, Reference, and Base--based on multi-modal conditions and diffusion timesteps: (i) a Mask Repaint module first refines coarse user-defined masks for precise spatial guidance; (ii) the router applies sparse top-K selection to dynamically allocate computation to the most relevant experts; (iii) a Latent Mixture module subsequently fuses expert outputs, coherently integrating semantic, spatial, and stylistic information to the base images. Experiments validate CARE-Edit's strong performance on contextual editing tasks, including erasure, replacement, text-driven edits, and style transfer. Empirical analysis further reveals task-specific behavior of specialized experts, showcasing the importance of dynamic, condition-aware processing to mitigate multi-condition conflicts.