Mapping the Mind of an Instruction-based Image Editing using SMILE

TL;DR

This paper introduces SMILE, a model-agnostic interpretability method that visualizes how textual instructions influence image editing models, enhancing transparency and trustworthiness in critical applications.

Contribution

The paper presents SMILE, a novel interpretability approach providing localized visual explanations for instruction-based image editing models, improving transparency and reliability.

Findings

SMILE produces effective heatmaps showing textual influence.

It improves interpretability and trust in image editing models.

Potential applications in healthcare and autonomous driving.

Abstract

Despite recent advancements in Instruct-based Image Editing models for generating high-quality images, they are known as black boxes and a significant barrier to transparency and user trust. To solve this issue, we introduce SMILE (Statistical Model-agnostic Interpretability with Local Explanations), a novel model-agnostic for localized interpretability that provides a visual heatmap to clarify the textual elements' influence on image-generating models. We applied our method to various Instruction-based Image Editing models like Pix2Pix, Image2Image-turbo and Diffusers-Inpaint and showed how our model can improve interpretability and reliability. Also, we use stability, accuracy, fidelity, and consistency metrics to evaluate our method. These findings indicate the exciting potential of model-agnostic interpretability for reliability and trustworthiness in critical applications such as healthcare and autonomous driving while encouraging additional investigation into the significance of interpretability in enhancing dependable image editing models.