Liars' Bench: Evaluating Lie Detectors for Language Models

TL;DR

This paper introduces LIARS' BENCH, a comprehensive testbed with over 72,000 examples of lies and honest responses from language models, revealing limitations of current lie detection techniques across diverse scenarios.

Contribution

The paper presents LIARS' BENCH, a large and diverse benchmark for evaluating lie detection methods in language models, highlighting their shortcomings and guiding future improvements.

Findings

Existing lie detection techniques often fail on certain lie types.

Detection accuracy drops when the transcript does not reveal the lie.

LIARS' BENCH exposes limitations of current methods across diverse settings.

Abstract

Prior work has introduced techniques for detecting when large language models (LLMs) lie, that is, generate statements they believe are false. However, these techniques are typically validated in narrow settings that do not capture the diverse lies LLMs can generate. We introduce LIARS' BENCH, a testbed consisting of 72,863 examples of lies and honest responses generated by four open-weight models across seven datasets. Our settings capture qualitatively different types of lies and vary along two dimensions: the model's reason for lying and the object of belief targeted by the lie. Evaluating three black- and white-box lie detection techniques on LIARS' BENCH, we find that existing techniques systematically fail to identify certain types of lies, especially in settings where it's not possible to determine whether the model lied from the transcript alone. Overall, LIARS' BENCH reveals limitations in prior techniques and provides a practical testbed for guiding progress in lie detection.