Deception Detection and Remote Physiological Monitoring: A Dataset and Baseline Experimental Results

TL;DR

This paper introduces the DDPM dataset, a comprehensive multimodal collection for deception detection and remote physiological monitoring, along with baseline experimental results demonstrating its potential for future research.

Contribution

The paper provides the first multimodal dataset with synchronized video, audio, and physiological data in an interview deception scenario, and offers initial baseline results.

Findings

Random accuracy for micro-expressions as deception indicators.

Saccades show statistically significant responses.

Heart rate can be estimated remotely with low error.

Abstract

We present the Deception Detection and Physiological Monitoring (DDPM) dataset and initial baseline results on this dataset. Our application context is an interview scenario in which the interviewee attempts to deceive the interviewer on selected responses. The interviewee is recorded in RGB, near-infrared, and long-wave infrared, along with cardiac pulse, blood oxygenation, and audio. After collection, data were annotated for interviewer/interviewee, curated, ground-truthed, and organized into train / test parts for a set of canonical deception detection experiments. Baseline experiments found random accuracy for micro-expressions as an indicator of deception, but that saccades can give a statistically significant response. We also estimated subject heart rates from face videos (remotely) with a mean absolute error as low as 3.16 bpm. The database contains almost 13 hours of recordings of 70 subjects, and over 8 million visible-light, near-infrared, and thermal video frames, along with appropriate meta, audio and pulse oximeter data. To our knowledge, this is the only collection offering recordings of five modalities in an interview scenario that can be used in both deception detection and remote photoplethysmography research.