Tutor-Student Reinforcement Learning: A Dynamic Curriculum for Robust Deepfake Detection

TL;DR

This paper introduces a Tutor-Student Reinforcement Learning framework that dynamically optimizes training curriculum for deepfake detection, leading to improved generalization and robustness.

Contribution

It proposes a novel RL-based curriculum learning method with a Tutor agent guiding a deepfake detector during training, which is a new approach in this domain.

Findings

Adaptive curriculum improves detection accuracy on unseen deepfake techniques.

The Tutor effectively prioritizes hard-but-learnable samples during training.

Code implementation is publicly available at the provided GitHub link.

Abstract

Standard supervised training for deepfake detection treats all samples with uniform importance, which can be suboptimal for learning robust and generalizable features. In this work, we propose a novel Tutor-Student Reinforcement Learning (TSRL) framework to dynamically optimize the training curriculum. Our method models the training process as a Markov Decision Process where a ``Tutor'' agent learns to guide a ``Student'' (the deepfake detector). The Tutor, implemented as a Proximal Policy Optimization (PPO) agent, observes a rich state representation for each training sample, encapsulating not only its visual features but also its historical learning dynamics, such as EMA loss and forgetting counts. Based on this state, the Tutor takes an action by assigning a continuous weight (0-1) to the sample's loss, thereby dynamically re-weighting the training batch. The Tutor is rewarded based on the Student's immediate performance change, specifically rewarding transitions from incorrect to correct predictions. This strategy encourages the Tutor to learn a curriculum that prioritizes high-value samples, such as hard-but-learnable examples, leading to a more efficient and effective training process. We demonstrate that this adaptive curriculum improves the Student's generalization capabilities against unseen manipulation techniques compared to traditional training methods. Code is available at https://github.com/wannac1/TSRL.