EditScout: Locating Forged Regions from Diffusion-based Edited Images with Multimodal LLM

TL;DR

This paper introduces EditScout, a novel framework that uses multimodal Large Language Models to effectively locate forged regions in diffusion-based edited images, surpassing existing methods especially on unseen data.

Contribution

The paper presents a new multimodal LLM-based approach for localizing diffusion-model forgeries, addressing limitations of traditional forensic techniques.

Findings

Outperforms previous methods on multiple datasets

Achieves higher mIoU and F1-score metrics

Excels on unseen diffusion-based edits in PerfBrush dataset

Abstract

Image editing technologies are tools used to transform, adjust, remove, or otherwise alter images. Recent research has significantly improved the capabilities of image editing tools, enabling the creation of photorealistic and semantically informed forged regions that are nearly indistinguishable from authentic imagery, presenting new challenges in digital forensics and media credibility. While current image forensic techniques are adept at localizing forged regions produced by traditional image manipulation methods, current capabilities struggle to localize regions created by diffusion-based techniques. To bridge this gap, we present a novel framework that integrates a multimodal Large Language Model (LLM) for enhanced reasoning capabilities to localize tampered regions in images produced by diffusion model-based editing methods. By leveraging the contextual and semantic strengths of LLMs, our framework achieves promising results on MagicBrush, AutoSplice, and PerfBrush (novel diffusion-based dataset) datasets, outperforming previous approaches in mIoU and F1-score metrics. Notably, our method excels on the PerfBrush dataset, a self-constructed test set featuring previously unseen types of edits. Here, where traditional methods typically falter, achieving markedly low scores, our approach demonstrates promising performance.