Generalized Zero and Few-Shot Transfer for Facial Forgery Detection

TL;DR

This paper introduces Deep Distribution Transfer, a novel transfer learning method that enhances facial forgery detection across unseen manipulation techniques and datasets, significantly improving zero and few-shot transfer performance.

Contribution

The paper presents a new mixture model-based loss and spatial mixup augmentation for effective zero and few-shot transfer in facial forgery detection, outperforming existing methods.

Findings

Improves zero-shot accuracy by 4.88% over baselines.

Enhances few-shot accuracy by 8.38% compared to state-of-the-art.

Effective domain transfer with limited target samples.

Abstract

We propose Deep Distribution Transfer(DDT), a new transfer learning approach to address the problem of zero and few-shot transfer in the context of facial forgery detection. We examine how well a model (pre-)trained with one forgery creation method generalizes towards a previously unseen manipulation technique or different dataset. To facilitate this transfer, we introduce a new mixture model-based loss formulation that learns a multi-modal distribution, with modes corresponding to class categories of the underlying data of the source forgery method. Our core idea is to first pre-train an encoder neural network, which maps each mode of this distribution to the respective class labels, i.e., real or fake images in the source domain by minimizing wasserstein distance between them. In order to transfer this model to a new domain, we associate a few target samples with one of the previously trained modes. In addition, we propose a spatial mixup augmentation strategy that further helps generalization across domains. We find this learning strategy to be surprisingly effective at domain transfer compared to a traditional classification or even state-of-the-art domain adaptation/few-shot learning methods. For instance, compared to the best baseline, our method improves the classification accuracy by 4.88% for zero-shot and by 8.38% for the few-shot case transferred from the FaceForensics++ to Dessa dataset.