DM-Align: Leveraging the Power of Natural Language Instructions to Make Changes to Images

TL;DR

DM-Align is a novel image editing model that uses natural language instructions and explicit word alignments to improve control, transparency, and preservation of image details during editing.

Contribution

It introduces a new method that explicitly reasons about image parts to alter or preserve, enhancing control and explainability in text-based image editing.

Findings

Outperforms state-of-the-art baselines in image editing quality

Better preserves background and handles long instructions

Demonstrates superior quantitative and qualitative results

Abstract

Text-based semantic image editing assumes the manipulation of an image using a natural language instruction. Although recent works are capable of generating creative and qualitative images, the problem is still mostly approached as a black box sensitive to generating unexpected outputs. Therefore, we propose a novel model to enhance the text-based control of an image editor by explicitly reasoning about which parts of the image to alter or preserve. It relies on word alignments between a description of the original source image and the instruction that reflects the needed updates, and the input image. The proposed Diffusion Masking with word Alignments (DM-Align) allows the editing of an image in a transparent and explainable way. It is evaluated on a subset of the Bison dataset and a self-defined dataset dubbed Dream. When comparing to state-of-the-art baselines, quantitative and qualitative results show that DM-Align has superior performance in image editing conditioned on language instructions, well preserves the background of the image and can better cope with long text instructions.