Mapping the Mind of an Instruction-based Image Editing using SMILE
Zeinab Dehghani, Koorosh Aslansefat, Adil Khan, Ad\'in, Ram\'irez Rivera, Franky George, Muhammad Khalid

TL;DR
This paper introduces SMILE, a model-agnostic interpretability method that visualizes how textual instructions influence image editing models, enhancing transparency and trustworthiness in critical applications.
Contribution
The paper presents SMILE, a novel interpretability approach providing localized visual explanations for instruction-based image editing models, improving transparency and reliability.
Findings
SMILE produces effective heatmaps showing textual influence.
It improves interpretability and trust in image editing models.
Potential applications in healthcare and autonomous driving.
Abstract
Despite recent advancements in Instruct-based Image Editing models for generating high-quality images, they are known as black boxes and a significant barrier to transparency and user trust. To solve this issue, we introduce SMILE (Statistical Model-agnostic Interpretability with Local Explanations), a novel model-agnostic for localized interpretability that provides a visual heatmap to clarify the textual elements' influence on image-generating models. We applied our method to various Instruction-based Image Editing models like Pix2Pix, Image2Image-turbo and Diffusers-Inpaint and showed how our model can improve interpretability and reliability. Also, we use stability, accuracy, fidelity, and consistency metrics to evaluate our method. These findings indicate the exciting potential of model-agnostic interpretability for reliability and trustworthiness in critical applications such as…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsOpen Education and E-Learning
Methods*Communicated@Fast*How Do I Communicate to Expedia? · Concatenated Skip Connection · Convolution · Sigmoid Activation · HuMan(Expedia)||How do I get a human at Expedia? · PatchGAN · Dropout · Batch Normalization · Pix2Pix · Heatmap
