FlexEdit: Marrying Free-Shape Masks to VLLM for Flexible Image Editing

TL;DR

FlexEdit introduces a novel end-to-end image editing approach that combines free-shape masks with language instructions, utilizing a vision large language model and a mask enhancement adapter to improve editing accuracy and user-friendliness.

Contribution

We propose FlexEdit, which effectively integrates free-shape masks with language instructions using a VLLM and a new Mask Enhance Adapter for improved image editing performance.

Findings

Achieves state-of-the-art results on LLM-based image editing tasks.

Introduces FSMI-Edit benchmark with 8 free-shape mask types.

Demonstrates effectiveness of simple prompting techniques.

Abstract

Combining Vision Large Language Models (VLLMs) with diffusion models offers a powerful method for executing image editing tasks based on human language instructions. However, language instructions alone often fall short in accurately conveying user requirements, particularly when users want to add, replace elements in specific areas of an image. Luckily, masks can effectively indicate the exact locations or elements to be edited, while they require users to precisely draw the shapes at the desired locations, which is highly user-unfriendly. To address this, we propose FlexEdit, an end-to-end image editing method that leverages both free-shape masks and language instructions for Flexible Editing. Our approach employs a VLLM in comprehending the image content, mask, and user instructions. Additionally, we introduce the Mask Enhance Adapter (MEA) that fuses the embeddings of the VLLM with the image data, ensuring a seamless integration of mask information and model output embeddings. Furthermore, we construct FSMI-Edit, a benchmark specifically tailored for free-shape mask, including 8 types of free-shape mask. Extensive experiments show that our method achieves state-of-the-art (SOTA) performance in LLM-based image editing, and our simple prompting technique stands out in its effectiveness. The code and data can be found at https://github.com/A-new-b/flex_edit.