SAGI: Semantically Aligned and Uncertainty Guided AI Image Inpainting

TL;DR

SAGI introduces a pipeline that enhances AI image inpainting by aligning generated images with human perception and evaluating their realism, leading to improved quality, aesthetics, and forensic detection capabilities.

Contribution

The paper presents SAGI, a novel, model-agnostic pipeline that improves semantic alignment and uncertainty estimation in AI image inpainting, and introduces the SAGI Dataset for benchmarking.

Findings

Semantic alignment improves image quality and aesthetics.

Uncertainty guidance reduces human error in distinguishing real from inpainted images.

Using SAGI-D enhances forensic detection performance.

Abstract

Recent advancements in generative AI have made text-guided image inpainting - adding, removing, or altering image regions using textual prompts - widely accessible. However, generating semantically correct photorealistic imagery, typically requires carefully-crafted prompts and iterative refinement by evaluating the realism of the generated content - tasks commonly performed by humans. To automate the generative process, we propose Semantically Aligned and Uncertainty Guided AI Image Inpainting (SAGI), a model-agnostic pipeline, to sample prompts from a distribution that closely aligns with human perception and to evaluate the generated content and discard instances that deviate from such a distribution, which we approximate using pretrained large language models and vision-language models. By applying this pipeline on multiple state-of-the-art inpainting models, we create the SAGI Dataset (SAGI-D), currently the largest and most diverse dataset of AI-generated inpaintings, comprising over 95k inpainted images and a human-evaluated subset. Our experiments show that semantic alignment significantly improves image quality and aesthetics, while uncertainty guidance effectively identifies realistic manipulations - human ability to distinguish inpainted images from real ones drops from 74% to 35% in terms of accuracy, after applying our pipeline. Moreover, using SAGI-D for training several image forensic approaches increases in-domain detection performance on average by 37.4% and out-of-domain generalization by 26.1% in terms of IoU, also demonstrating its utility in countering malicious exploitation of generative AI. Code and dataset are available at https://mever-team.github.io/SAGI/