Unraveling Hidden Representations: A Multi-Modal Layer Analysis for Better Synthetic Content Forensics

TL;DR

This paper proposes a multi-modal layer analysis approach using large pre-trained models to detect synthetic media, achieving robust, cross-modal fake detection with efficient training and strong results in images and audio.

Contribution

It introduces a universal fake detection method leveraging latent codes of pre-trained multi-modal models, outperforming existing classifiers across multiple data domains.

Findings

Linear classifiers on latent codes achieve state-of-the-art results.

Method is effective in few-shot detection scenarios.

Approach surpasses or matches strong baseline methods in images and audio.

Abstract

Generative models achieve remarkable results in multiple data domains, including images and texts, among other examples. Unfortunately, malicious users exploit synthetic media for spreading misinformation and disseminating deepfakes. Consequently, the need for robust and stable fake detectors is pressing, especially when new generative models appear everyday. While the majority of existing work train classifiers that discriminate between real and fake information, such tools typically generalize only within the same family of generators and data modalities, yielding poor results on other generative classes and data domains. Towards a universal classifier, we propose the use of large pre-trained multi-modal models for the detection of generative content. Effectively, we show that the latent code of these models naturally captures information discriminating real from fake. Building on this observation, we demonstrate that linear classifiers trained on these features can achieve state-of-the-art results across various modalities, while remaining computationally efficient, fast to train, and effective even in few-shot settings. Our work primarily focuses on fake detection in audio and images, achieving performance that surpasses or matches that of strong baseline methods.