Multiverse Through Deepfakes: The MultiFakeVerse Dataset of Person-Centric Visual and Conceptual Manipulations

TL;DR

This paper introduces MultiFakeVerse, a large-scale, person-centric deepfake dataset generated via vision-language models, enabling semantic and context-aware manipulations to challenge current detection methods and improve deepfake understanding.

Contribution

The creation of MultiFakeVerse, a novel large-scale dataset with semantic, context-aware deepfake manipulations driven by vision-language models, addressing a key gap in existing benchmarks.

Findings

State-of-the-art detection models struggle with subtle manipulations.

Humans also find it difficult to detect these semantic deepfakes.

The dataset enables research on reasoning-based deepfake detection.

Abstract

The rapid advancement of GenAI technology over the past few years has significantly contributed towards highly realistic deepfake content generation. Despite ongoing efforts, the research community still lacks a large-scale and reasoning capability driven deepfake benchmark dataset specifically tailored for person-centric object, context and scene manipulations. In this paper, we address this gap by introducing MultiFakeVerse, a large scale person-centric deepfake dataset, comprising 845,286 images generated through manipulation suggestions and image manipulations both derived from vision-language models (VLM). The VLM instructions were specifically targeted towards modifications to individuals or contextual elements of a scene that influence human perception of importance, intent, or narrative. This VLM-driven approach enables semantic, context-aware alterations such as modifying actions, scenes, and human-object interactions rather than synthetic or low-level identity swaps and region-specific edits that are common in existing datasets. Our experiments reveal that current state-of-the-art deepfake detection models and human observers struggle to detect these subtle yet meaningful manipulations. The code and dataset are available on \href{https://github.com/Parul-Gupta/MultiFakeVerse}{GitHub}.