Characterizing the temporal dynamics of universal speech representations for generalizable deepfake detection

TL;DR

This paper investigates the temporal dynamics of universal speech representations to improve deepfake detection, demonstrating that understanding these dynamics enhances generalization to unseen attack methods.

Contribution

It introduces a novel method to characterize the long-term temporal dynamics of speech representations, improving deepfake detection across unseen generative models.

Findings

Representation dynamics are similar across different generative models.

The proposed method improves detection of unseen deepfake attacks.

Significant performance gains over benchmark methods.

Abstract

Existing deepfake speech detection systems lack generalizability to unseen attacks (i.e., samples generated by generative algorithms not seen during training). Recent studies have explored the use of universal speech representations to tackle this issue and have obtained inspiring results. These works, however, have focused on innovating downstream classifiers while leaving the representation itself untouched. In this study, we argue that characterizing the long-term temporal dynamics of these representations is crucial for generalizability and propose a new method to assess representation dynamics. Indeed, we show that different generative models generate similar representation dynamics patterns with our proposed method. Experiments on the ASVspoof 2019 and 2021 datasets validate the benefits of the proposed method to detect deepfakes from methods unseen during training, significantly improving on several benchmark methods.