Judgment-of-Thought Prompting: A Courtroom-Inspired Framework for Binary Logical Reasoning with Large Language Models

TL;DR

The paper introduces Judgment of Thought (JoT), a multi-agent prompting framework inspired by courtroom roles, which significantly improves binary logical reasoning accuracy in large language models through debate and systematic evaluation.

Contribution

JoT is a novel multi-agent prompting approach that models courtroom roles to enhance logical reasoning in large language models, outperforming existing methods on key benchmarks.

Findings

Achieves 98% accuracy on Boolean expressions

Outperforms existing prompting approaches on benchmarks

Ablation studies confirm the importance of each role and feedback mechanisms

Abstract

This paper proposes a novel prompting approach, Judgment of Thought (JoT), specifically tailored for binary logical reasoning tasks. Despite advances in prompt engineering, existing approaches still face limitations in handling complex logical reasoning tasks. To address these issues, JoT introduces a multi-agent approach with three specialized roles$\unicode{x2010}$$\unicode{x2010}$$\unicode{x2010}$lawyer, prosecutor, and judge$\unicode{x2010}$$\unicode{x2010}$$\unicode{x2010}$where a high-level model acts as the judge, and lower-level models serve as lawyer and prosecutor to systematically debate and evaluate arguments. Experimental evaluations on benchmarks such as BigBenchHard and Winogrande demonstrate JoT's superior performance compared to existing prompting approaches, achieving notable improvements, including 98\% accuracy in Boolean expressions. Also, our ablation studies validate the critical contribution of each role, iterative refinement loops, and feedback mechanisms. Consequently, JoT significantly enhances accuracy, reliability, and consistency in binary reasoning tasks and shows potential for practical applications.