EXIF as Language: Learning Cross-Modal Associations Between Images and Camera Metadata

TL;DR

This paper introduces a multimodal embedding model that learns to associate image patches with camera metadata, enabling improved performance on image forensics and calibration tasks, including zero-shot splicing localization.

Contribution

It presents a novel approach to learn cross-modal associations between images and EXIF metadata using a transformer-based model, outperforming existing features.

Findings

Significantly better performance on forensics and calibration tasks.

Effective zero-shot localization of spliced regions.

Outperforms prior self-supervised and supervised methods.

Abstract

We learn a visual representation that captures information about the camera that recorded a given photo. To do this, we train a multimodal embedding between image patches and the EXIF metadata that cameras automatically insert into image files. Our model represents this metadata by simply converting it to text and then processing it with a transformer. The features that we learn significantly outperform other self-supervised and supervised features on downstream image forensics and calibration tasks. In particular, we successfully localize spliced image regions "zero shot" by clustering the visual embeddings for all of the patches within an image.