Benchmarking Cross-Domain Audio-Visual Deception Detection

TL;DR

This paper introduces the first cross-domain benchmark for audio-visual deception detection, evaluating generalization across scenarios and proposing methods to improve robustness using multi-source training and novel fusion techniques.

Contribution

It presents a comprehensive cross-domain benchmark, explores domain sampling strategies, and introduces the MM-IDGM algorithm and Attention-Mixer fusion to enhance generalization.

Findings

Multi-source domain training improves detection accuracy.

Proposed methods outperform baseline models in cross-domain scenarios.

Benchmark facilitates future research in real-world deception detection.

Abstract

Automated deception detection is crucial for assisting humans in accurately assessing truthfulness and identifying deceptive behavior. Conventional contact-based techniques, like polygraph devices, rely on physiological signals to determine the authenticity of an individual's statements. Nevertheless, recent developments in automated deception detection have demonstrated that multimodal features derived from both audio and video modalities may outperform human observers on publicly available datasets. Despite these positive findings, the generalizability of existing audio-visual deception detection approaches across different scenarios remains largely unexplored. To close this gap, we present the first cross-domain audio-visual deception detection benchmark, that enables us to assess how well these methods generalize for use in real-world scenarios. We used widely adopted audio and visual features and different architectures for benchmarking, comparing single-to-single and multi-to-single domain generalization performance. To further exploit the impacts using data from multiple source domains for training, we investigate three types of domain sampling strategies, including domain-simultaneous, domain-alternating, and domain-by-domain for multi-to-single domain generalization evaluation. We also propose an algorithm to enhance the generalization performance by maximizing the gradient inner products between modality encoders, named ``MM-IDGM". Furthermore, we proposed the Attention-Mixer fusion method to improve performance, and we believe that this new cross-domain benchmark will facilitate future research in audio-visual deception detection.