I Dream My Painting: Connecting MLLMs and Diffusion Models via Prompt Generation for Text-Guided Multi-Mask Inpainting

TL;DR

This paper introduces a novel multi-mask inpainting framework that uses multimodal LLMs to generate prompts for multiple regions, combined with diffusion models for detailed, text-guided image restoration.

Contribution

It proposes a new multi-mask inpainting task and a method to automatically generate prompts using fine-tuned multimodal LLMs, enhancing inpainting accuracy and creativity.

Findings

Effective multi-mask inpainting on artistic images

Generated prompts improve inpainting quality

Method outperforms existing diffusion-based inpainting approaches

Abstract

Inpainting focuses on filling missing or corrupted regions of an image to blend seamlessly with its surrounding content and style. While conditional diffusion models have proven effective for text-guided inpainting, we introduce the novel task of multi-mask inpainting, where multiple regions are simultaneously inpainted using distinct prompts. Furthermore, we design a fine-tuning procedure for multimodal LLMs, such as LLaVA, to generate multi-mask prompts automatically using corrupted images as inputs. These models can generate helpful and detailed prompt suggestions for filling the masked regions. The generated prompts are then fed to Stable Diffusion, which is fine-tuned for the multi-mask inpainting problem using rectified cross-attention, enforcing prompts onto their designated regions for filling. Experiments on digitized paintings from WikiArt and the Densely Captioned Images dataset demonstrate that our pipeline delivers creative and accurate inpainting results. Our code, data, and trained models are available at https://cilabuniba.github.io/i-dream-my-painting.