Image Provenance Analysis via Graph Encoding with Vision Transformer

TL;DR

This paper presents a novel deep learning framework using graph encoding and vision transformers to improve image provenance analysis, capturing both local manipulations and global transformations for more accurate relationship graph construction.

Contribution

The proposed method introduces a patch attention mechanism and aligns training with graph construction, enhancing generalizability and structural understanding in image provenance analysis.

Findings

Outperforms previous methods in provenance graph accuracy

Effectively captures local and global image manipulation cues

Demonstrates robustness across different image datasets

Abstract

Recent advances in AI-powered image editing tools have significantly lowered the barrier to image modification, raising pressing security concerns those related to spreading misinformation and disinformation on social platforms. Image provenance analysis is crucial in this context, as it identifies relevant images within a database and constructs a relationship graph by mining hidden manipulation and transformation cues, thereby providing concrete evidence chains. This paper introduces a novel end-to-end deep learning framework designed to explore the structural information of provenance graphs. Our proposed method distinguishes from previous approaches in two main ways. First, unlike earlier methods that rely on prior knowledge and have limited generalizability, our framework relies upon a patch attention mechanism to capture image provenance clues for local manipulations and global transformations, thereby enhancing graph construction performance. Second, while previous methods primarily focus on identifying tampering traces only between image pairs, they often overlook the hidden information embedded in the topology of the provenance graph. Our approach aligns the model training objectives with the final graph construction task, incorporating the overall structural information of the graph into the training process. We integrate graph structure information with the attention mechanism, enabling precise determination of the direction of transformation. Experimental results show the superiority of the proposed method over previous approaches, underscoring its effectiveness in addressing the challenges of image provenance analysis.