MIND-Edit: MLLM Insight-Driven Editing via Language-Vision Projection

TL;DR

MIND-Edit is an innovative image editing framework that combines pretrained diffusion models with multimodal large language models to improve semantic accuracy and visual coherence in complex editing tasks.

Contribution

It introduces a novel end-to-end approach that leverages MLLM's visual understanding and semantic reasoning for more precise and semantically aligned image edits.

Findings

Outperforms state-of-the-art methods in quantitative metrics.

Achieves more visually coherent edits in complex scenarios.

Enhances instruction interpretation accuracy.

Abstract

Recent advances in AI-generated content (AIGC) have significantly accelerated image editing techniques, driving increasing demand for diverse and fine-grained edits. Despite these advances, existing image editing methods still face challenges in achieving high precision and semantic accuracy in complex scenarios. Recent studies address this issue by incorporating multimodal large language models (MLLMs) into image editing pipelines. However, current MLLM-based methods mainly rely on interpreting textual instructions, leaving the intrinsic visual understanding of large models largely unexplored, thus resulting in insufficient alignment between textual semantics and visual outcomes. To overcome these limitations, we propose MIND-Edit, an end-to-end image-editing framework integrating pretrained diffusion model with MLLM. MIND-Edit introduces two complementary strategies: (1) a text instruction optimization strategy that clarifies ambiguous user instructions based on semantic reasoning from the MLLM, and (2) an MLLM insight-driven editing strategy that explicitly leverages the intrinsic visual understanding capability of the MLLM to infer editing intent and guide the diffusion process via generated visual embeddings. Furthermore, we propose a joint training approach to effectively integrate both strategies, allowing them to reinforce each other for more accurate instruction interpretation and visually coherent edits aligned with user intent. Extensive experiments demonstrate that MIND-Edit outperforms state-of-the-art image editing methods in both quantitative metrics and visual quality, particularly under complex and challenging scenarios.