Hidden in Plain Sight: Evaluation of the Deception Detection Capabilities of LLMs in Multimodal Settings

TL;DR

This paper evaluates the deception detection capabilities of large language and multimodal models across various datasets and experimental setups, revealing their strengths and limitations in multimodal deception detection.

Contribution

It provides a comprehensive analysis of LLMs and LMMs in deception detection, comparing different models, datasets, and prompting strategies, and highlights the effectiveness of fine-tuned LLMs and challenges faced by LMMs.

Findings

Fine-tuned LLMs achieve state-of-the-art textual deception detection performance.

LMMs struggle to utilize cross-modal cues effectively.

Prompting strategies significantly impact detection accuracy.

Abstract

Detecting deception in an increasingly digital world is both a critical and challenging task. In this study, we present a comprehensive evaluation of the automated deception detection capabilities of Large Language Models (LLMs) and Large Multimodal Models (LMMs) across diverse domains. We assess the performance of both open-source and commercial LLMs on three distinct datasets: real life trial interviews (RLTD), instructed deception in interpersonal scenarios (MU3D), and deceptive reviews (OpSpam). We systematically analyze the effectiveness of different experimental setups for deception detection, including zero-shot and few-shot approaches with random or similarity-based in-context example selection. Our results show that fine-tuned LLMs achieve state-of-the-art performance on textual deception detection tasks, while LMMs struggle to fully leverage cross-modal cues. Additionally, we analyze the impact of auxiliary features, such as non-verbal gestures and video summaries, and examine the effectiveness of different prompting strategies, including direct label generation and chain-of-thought reasoning. Our findings provide key insights into how LLMs process and interpret deceptive cues across modalities, highlighting their potential and limitations in real-world deception detection applications.