Dynamic Emotion and Personality Profiling for Multimodal Deception Detection

TL;DR

This paper introduces a novel multimodal framework for deception detection that incorporates dynamic emotion and personality annotations, significantly improving detection accuracy across multiple datasets.

Contribution

It proposes a new annotation scheme, a reliability-weighted fusion framework, and a joint detection model for deception, emotion, and personality.

Findings

F1 score for deception detection increased by 2.53%.

F1 score for emotion detection increased by 2.66%.

F1 score for personality detection increased by 9.30%.

Abstract

Deception detection is of great significance for ensuring information security and conducting public opinion analysis, with personality factors and emotion cues playing a critical role. However, existing methods lack sample-level dynamic annotations for emotions and personality.In this paper, we propose an innovative multi-model multi-prompt annotation scheme and a strict label quality evaluation standard, and establish a multimodal joint detection dataset DDEP for deception, emotion, and personality. Meanwhile, we propose Rel-DDEP, an adaptive reliability-weighted fusion framework. Our framework quantifies uncertainty by mapping modal features to a high-dimensional Gaussian distribution space. It then performs reliability-weighted fusion and incorporates an alignment module and a sorting constraint module to achieve joint detection of deception, emotion, and personality. Experimental results on the MDPE and DDEP datasets show that our Rel-DDEP significantly outperforms the existing state-of-the-art baseline models in three tasks. The F1 score of the deception detection increases by 2.53%, that of the emotion detection increases by 2.66%, and that of the personality detection increases by 9.30%. The experiments fully verify the necessity of annotating dynamic emotion and personality labels for each sample and the effectiveness of reliability-weighted fusion.