AFFAKT: A Hierarchical Optimal Transport based Method for Affective Facial Knowledge Transfer in Video Deception Detection

TL;DR

This paper introduces AFFAKT, a hierarchical optimal transport-based method that transfers facial expression knowledge to improve video deception detection, addressing data scarcity and enhancing interpretability.

Contribution

The paper proposes a novel hierarchical optimal transport-based framework for effective knowledge transfer from facial expressions to deception detection, incorporating correlation retention and sample-specific fine-tuning.

Findings

Outperforms existing methods on two deception datasets.

Shows high correlation between deception and negative expressions.

Provides interpretability aligning with psychological theories.

Abstract

The scarcity of high-quality large-scale labeled datasets poses a huge challenge for employing deep learning models in video deception detection. To address this issue, inspired by the psychological theory on the relation between deception and expressions, we propose a novel method called AFFAKT in this paper, which enhances the classification performance by transferring useful and correlated knowledge from a large facial expression dataset. Two key challenges in knowledge transfer arise: 1) \textit{how much} knowledge of facial expression data should be transferred and 2) \textit{how to} effectively leverage transferred knowledge for the deception classification model during inference. Specifically, the optimal relation mapping between facial expression classes and deception samples is firstly quantified using proposed H-OTKT module and then transfers knowledge from the facial expression dataset to deception samples. Moreover, a correlation prototype within another proposed module SRKB is well designed to retain the invariant correlations between facial expression classes and deception classes through momentum updating. During inference, the transferred knowledge is fine-tuned with the correlation prototype using a sample-specific re-weighting strategy. Experimental results on two deception detection datasets demonstrate the superior performance of our proposed method. The interpretability study reveals high associations between deception and negative affections, which coincides with the theory in psychology.