PU-Lie: Lightweight Deception Detection in Imbalanced Diplomatic Dialogues via Positive-Unlabeled Learning

TL;DR

This paper presents PU-Lie, a lightweight deception detection model tailored for imbalanced diplomatic dialogues, leveraging positive-unlabeled learning, linguistic features, and speaker information to improve detection accuracy and interpretability.

Contribution

The paper introduces PU-Lie, a novel deception detection approach that effectively handles extreme class imbalance using PU learning and interpretable features, with significantly fewer trainable parameters.

Findings

Achieves a macro F1 of 0.60 on the Diplomacy dataset.

Reduces trainable parameters by over 650 times.

Demonstrates the effectiveness of PU learning in deception detection.

Abstract

Detecting deception in strategic dialogues is a complex and high-stakes task due to the subtlety of language and extreme class imbalance between deceptive and truthful communications. In this work, we revisit deception detection in the Diplomacy dataset, where less than 5% of messages are labeled deceptive. We introduce a lightweight yet effective model combining frozen BERT embeddings, interpretable linguistic and game-specific features, and a Positive-Unlabeled (PU) learning objective. Unlike traditional binary classifiers, PU-Lie is tailored for situations where only a small portion of deceptive messages are labeled, and the majority are unlabeled. Our model achieves a new best macro F1 of 0.60 while reducing trainable parameters by over 650x. Through comprehensive evaluations and ablation studies across seven models, we demonstrate the value of PU learning, linguistic interpretability, and speaker-aware representations. Notably, we emphasize that in this problem setting, accurately detecting deception is more critical than identifying truthful messages. This priority guides our choice of PU learning, which explicitly models the rare but vital deceptive class.