Propose and Rectify: A Forensics-Driven MLLM Framework for Image Manipulation Localization

TL;DR

This paper introduces a novel forensic framework combining semantic reasoning and forensic analysis to improve the detection and localization of manipulated regions in images, achieving state-of-the-art results.

Contribution

It proposes a Propose-Rectify framework that integrates a forensic-adapted LLaVA model with forensic feature analysis and enhanced segmentation to improve manipulation localization.

Findings

Achieves state-of-the-art detection accuracy.

Demonstrates robustness across diverse datasets.

Effectively combines semantic reasoning with forensic evidence.

Abstract

The increasing sophistication of image manipulation techniques demands robust forensic solutions that can both reliably detect alterations and precisely localize tampered regions. Recent Multimodal Large Language Models (MLLMs) show promise by leveraging world knowledge and semantic understanding for context-aware detection, yet they struggle with perceiving subtle, low-level forensic artifacts crucial for accurate manipulation localization. This paper presents a novel Propose-Rectify framework that effectively bridges semantic reasoning with forensic-specific analysis. In the proposal stage, our approach utilizes a forensic-adapted LLaVA model to generate initial manipulation analysis and preliminary localization of suspicious regions based on semantic understanding and contextual reasoning. In the rectification stage, we introduce a Forensics Rectification Module that systematically validates and refines these initial proposals through multi-scale forensic feature analysis, integrating technical evidence from several specialized filters. Additionally, we present an Enhanced Segmentation Module that incorporates critical forensic cues into SAM's encoded image embeddings, thereby overcoming inherent semantic biases to achieve precise delineation of manipulated regions. By synergistically combining advanced multimodal reasoning with established forensic methodologies, our framework ensures that initial semantic proposals are systematically validated and enhanced through concrete technical evidence, resulting in comprehensive detection accuracy and localization precision. Extensive experimental validation demonstrates state-of-the-art performance across diverse datasets with exceptional robustness and generalization capabilities.