LLMs are Superior Feedback Providers: Bootstrapping Reasoning for Lie Detection with Self-Generated Feedback

TL;DR

This paper introduces a bootstrapping framework where LLMs generate and refine feedback to improve lie detection in complex dialogues, achieving significant performance gains without training data.

Contribution

The paper presents a novel self-feedback bootstrapping method that enhances LLM reasoning for lie detection, outperforming zero-shot baselines and rivaling supervised methods.

Findings

39% improvement in lying-F1 score over zero-shot baseline

LLM-generated feedback surpasses professional human feedback quality

Achieves state-of-the-art results without training data

Abstract

Large Language Models (LLMs) excel at generating human-like dialogues and comprehending text. However, understanding the subtleties of complex exchanges in language remains a challenge. We propose a bootstrapping framework that leverages self-generated feedback to enhance LLM reasoning capabilities for lie detection. The framework consists of three stages: suggestion, feedback collection, and modification. In the suggestion stage, a cost-effective language model generates initial predictions based on game state and dialogue. The feedback-collection stage involves a language model providing feedback on these predictions. In the modification stage, a more advanced language model refines the initial predictions using the auto-generated feedback. We investigate the application of the proposed framework for detecting betrayal and deception in Diplomacy games, and compare it with feedback from professional human players. The LLM-generated feedback exhibits superior quality and significantly enhances the performance of the model. Our approach achieves a 39% improvement over the zero-shot baseline in lying-F1 without the need for any training data, rivaling state-of-the-art supervised learning results.