Beyond Spatial Frequency: Pixel-wise Temporal Frequency-based Deepfake Video Detection

TL;DR

This paper presents a novel deepfake video detection method that analyzes pixel-wise temporal frequency using Fourier transforms and attention mechanisms, improving detection of subtle temporal artifacts.

Contribution

It introduces a pixel-wise temporal frequency analysis combined with an attention proposal and transformer modules for enhanced deepfake detection.

Findings

Robust detection performance across diverse scenarios

Effective localization of temporal artifacts

Improved sensitivity to unnatural movements

Abstract

We introduce a deepfake video detection approach that exploits pixel-wise temporal inconsistencies, which traditional spatial frequency-based detectors often overlook. Traditional detectors represent temporal information merely by stacking spatial frequency spectra across frames, resulting in the failure to detect temporal artifacts in the pixel plane. Our approach performs a 1D Fourier transform on the time axis for each pixel, extracting features highly sensitive to temporal inconsistencies, especially in areas prone to unnatural movements. To precisely locate regions containing the temporal artifacts, we introduce an attention proposal module trained in an end-to-end manner. Additionally, our joint transformer module effectively integrates pixel-wise temporal frequency features with spatio-temporal context features, expanding the range of detectable forgery artifacts. Our framework represents a significant advancement in deepfake video detection, providing robust performance across diverse and challenging detection scenarios.