InstructAttribute: Fine-grained Object Attributes editing with Instruction

TL;DR

InstructAttribute is a novel instruction-tuned model that enables precise, fine-grained editing of object attributes like color and material in images, leveraging a new training-free framework and large language models for data curation.

Contribution

The paper introduces InstructAttribute, a new method for object attribute editing that combines a training-free framework with large language models for data generation, improving accuracy and structural preservation.

Findings

Outperforms existing instruction-based methods in attribute editing accuracy

Achieves a good balance between attribute modification and image structural integrity

Enables practical applications in product design, e-commerce, and virtual try-on

Abstract

Text-to-image (T2I) diffusion models are widely used in image editing due to their powerful generative capabilities. However, achieving fine-grained control over specific object attributes, such as color and material, remains a considerable challenge. Existing methods often fail to accurately modify these attributes or compromise structural integrity and overall image consistency. To fill this gap, we introduce Structure Preservation and Attribute Amplification (SPAA), a novel training-free framework that enables precise generation of color and material attributes for the same object by intelligently manipulating self-attention maps and cross-attention values within diffusion models. Building on SPAA, we integrate multi-modal large language models (MLLMs) to automate data curation and instruction generation. Leveraging this object attribute data collection engine, we construct the Attribute Dataset, encompassing a comprehensive range of colors and materials across diverse object categories. Using this generated dataset, we propose InstructAttribute, an instruction-tuned model that enables fine-grained and object-level attribute editing through natural language prompts. This capability holds significant practical implications for diverse fields, from accelerating product design and e-commerce visualization to enhancing virtual try-on experiences. Extensive experiments demonstrate that InstructAttribute outperforms existing instruction-based baselines, achieving a superior balance between attribute modification accuracy and structural preservation.