Enhancing Lie Detection Accuracy: A Comparative Study of Classic ML, CNN, and GCN Models using Audio-Visual Features

TL;DR

This study compares classic machine learning, CNN, and GCN models using audio-visual features to improve non-invasive lie detection accuracy, achieving up to 95.4% accuracy with a multimodal transformer approach.

Contribution

It introduces a multimodal transformer architecture combining audio, visual, and gesture features for enhanced lie detection, advancing beyond previous micro-expression based models.

Findings

CNN model achieved 95.4% accuracy

Multimodal features improved detection reliability

Further research needed for dataset quality and generalization

Abstract

Inaccuracies in polygraph tests often lead to wrongful convictions, false information, and bias, all of which have significant consequences for both legal and political systems. Recently, analyzing facial micro-expressions has emerged as a method for detecting deception; however, current models have not reached high accuracy and generalizability. The purpose of this study is to aid in remedying these problems. The unique multimodal transformer architecture used in this study improves upon previous approaches by using auditory inputs, visual facial micro-expressions, and manually transcribed gesture annotations, moving closer to a reliable non-invasive lie detection model. Visual and auditory features were extracted using the Vision Transformer and OpenSmile models respectively, which were then concatenated with the transcriptions of participants micro-expressions and gestures. Various models were trained for the classification of lies and truths using these processed and concatenated features. The CNN Conv1D multimodal model achieved an average accuracy of 95.4%. However, further research is still required to create higher-quality datasets and even more generalized models for more diverse applications.