IMAGHarmony: Controllable Image Editing with Consistent Object Quantity and Layout

TL;DR

IMAGHarmony is a novel diffusion-based image editing framework that preserves object counts and spatial layouts during multi-object scene modifications, ensuring structural and semantic consistency.

Contribution

It introduces a harmony-aware module and a preference-guided noise strategy, enabling efficient, structure-preserving multi-object image editing with minimal training data.

Findings

Outperforms existing methods in structural preservation and semantic accuracy.

Requires only 200 training images and 10.6M trainable parameters.

Provides a new benchmark, HarmonyBench, for evaluating multi-object editing.

Abstract

Despite advances in diffusion-based image editing, manipulating multi-object scenes remains challenging. Existing approaches often achieve semantic changes at the expense of structural consistency, failing to preserve exact object counts and spatial layouts without introducing unintended relocations or background modifications. To address this limitation, we introduce quantity-and-layout-consistent image editing (QL-Edit) to modify object semantics while maintaining the original instance cardinality and spatial layout. We propose IMAGHarmony, a parameter-efficient framework featuring a harmony-aware (HA) module that incorporates perception cues from the reference image into the diffusion process. This enables the model to jointly reason about object semantics, counts, and spatial positions for improved structural consistency. Furthermore, we introduce a preference-guided noise selection (PNS) strategy that identifies favorable initialization conditions, substantially improving generation stability in challenging multi-object scenarios. To support systematic evaluation, we construct HarmonyBench, a benchmark designed to measure semantic editing accuracy and structural consistency under quantity and layout constraints. Extensive experiments demonstrate that IMAGHarmony consistently outperforms existing methods in both structural preservation and semantic accuracy. Notably, our framework is highly efficient, requiring only 200 training images and 10.6M trainable parameters. Code, models, and data are available at \url{https://github.com/muzishen/IMAGHarmony}.