Does Fine-tuning by Reinforcement Learning Improve Generalization in Binary Speech Deepfake Detection?

TL;DR

This paper explores the use of reinforcement learning, specifically GRPO, for fine-tuning speech deepfake detectors, showing it enhances out-of-domain generalization compared to traditional supervised methods.

Contribution

It introduces RL-based fine-tuning with GRPO for speech deepfake detection, demonstrating improved out-of-domain performance over standard supervised fine-tuning.

Findings

GRPO-based fine-tuning improves out-of-domain detection accuracy.

Pure GRPO fine-tuning outperforms hybrid and supervised methods.

Negative rewards in GRPO may be crucial for generalization.

Abstract

Building speech deepfake detection models that are generalizable to unseen attacks remains a challenging problem. Although the field has shifted toward a pre-training and fine-tuning paradigm using speech foundation models, most approaches rely solely on supervised fine-tuning (SFT). Inspired by the field of large language models, wherein reinforcement learning (RL) is used for model fine-tuning, we investigate the impact of RL, specifically Group Relative Policy Optimization (GRPO). The results from experiments using multiple detectors and test sets indicate that pure GRPO-based fine-tuning improves performance on out-of-domain test sets while maintaining performance on target-domain test data. This approach outperforms both SFT-only and hybrid setups. Our ablation studies further suggest that the negative reward in GRPO may be a key factor in this improvement.