PROVES: Establishing Image Provenance using Semantic Signatures

TL;DR

PROVES introduces a semantic signing architecture to verify image provenance, focusing on identity and scene context, robust against common transformations and adversarial attacks, enhancing trust in visual data authenticity.

Contribution

The paper presents a novel semantic signing and verification architecture for image provenance, addressing robustness against transformations and adversarial manipulations.

Findings

Robust face provenance verification against black-box adversarial transformations.

Background verification improves significantly with adversarial training.

The approach effectively rejects adversarially perturbed images while accepting benign transformations.

Abstract

Modern AI tools, such as generative adversarial networks, have transformed our ability to create and modify visual data with photorealistic results. However, one of the deleterious side-effects of these advances is the emergence of nefarious uses in manipulating information in visual data, such as through the use of deep fakes. We propose a novel architecture for preserving the provenance of semantic information in images to make them less susceptible to deep fake attacks. Our architecture includes semantic signing and verification steps. We apply this architecture to verifying two types of semantic information: individual identities (faces) and whether the photo was taken indoors or outdoors. Verification accounts for a collection of common image transformation, such as translation, scaling, cropping, and small rotations, and rejects adversarial transformations, such as adversarially perturbed or, in the case of face verification, swapped faces. Experiments demonstrate that in the case of provenance of faces in an image, our approach is robust to black-box adversarial transformations (which are rejected) as well as benign transformations (which are accepted), with few false negatives and false positives. Background verification, on the other hand, is susceptible to black-box adversarial examples, but becomes significantly more robust after adversarial training.