Automated Deception Detection from Videos: Using End-to-End Learning Based High-Level Features and Classification Approaches

TL;DR

This paper presents a multimodal deep learning approach for automated deception detection from videos, combining facial cues, gaze, and head pose, evaluated across multiple datasets with promising results despite limited data.

Contribution

It introduces an end-to-end convolutional model for deception detection using video features and demonstrates the effectiveness of multimodal feature combination and selection.

Findings

Facial expressions outperform gaze and head pose in deception detection.

Combining modalities with feature selection improves accuracy.

Detection performance exceeds chance levels even in low-stake scenarios.

Abstract

Deception detection is an interdisciplinary field attracting researchers from psychology, criminology, computer science, and economics. We propose a multimodal approach combining deep learning and discriminative models for automated deception detection. Using video modalities, we employ convolutional end-to-end learning to analyze gaze, head pose, and facial expressions, achieving promising results compared to state-of-the-art methods. Due to limited training data, we also utilize discriminative models for deception detection. Although sequence-to-class approaches are explored, discriminative models outperform them due to data scarcity. Our approach is evaluated on five datasets, including a new Rolling-Dice Experiment motivated by economic factors. Results indicate that facial expressions outperform gaze and head pose, and combining modalities with feature selection enhances detection performance. Differences in expressed features across datasets emphasize the importance of scenario-specific training data and the influence of context on deceptive behavior. Cross-dataset experiments reinforce these findings. Despite the challenges posed by low-stake datasets, including the Rolling-Dice Experiment, deception detection performance exceeds chance levels. Our proposed multimodal approach and comprehensive evaluation shed light on the potential of automating deception detection from video modalities, opening avenues for future research.