Towards Logically Sound Natural Language Reasoning with Logic-Enhanced Language Model Agents

TL;DR

This paper introduces LELMA, a framework that combines language models with formal logic to improve the logical soundness and correctness of natural language reasoning in open-ended tasks.

Contribution

The paper proposes a novel framework integrating LLMs with formal logic for validation and refinement, addressing logical errors in natural language reasoning.

Findings

LELMA achieves high accuracy in error detection.

Self-refinement improves reasoning correctness, especially in GPT-4o.

Challenges remain in autoformalization accuracy and evaluating ambiguous tasks.

Abstract

Large language models (LLMs) are increasingly explored as general-purpose reasoners, particularly in agentic contexts. However, their outputs remain prone to mathematical and logical errors. This is especially challenging in open-ended tasks, where unstructured outputs lack explicit ground truth and may contain subtle inconsistencies. To address this issue, we propose Logic-Enhanced Language Model Agents (LELMA), a framework that integrates LLMs with formal logic to enable validation and refinement of natural language reasoning. LELMA comprises three components: an LLM-Reasoner, an LLM-Translator, and a Solver, and employs autoformalization to translate reasoning into logic representations, which are then used to assess logical validity. Using game-theoretic scenarios such as the Prisoner's Dilemma as testbeds, we highlight the limitations of both less capable (Gemini 1.0 Pro) and advanced (GPT-4o) models in generating logically sound reasoning. LELMA achieves high accuracy in error detection and improves reasoning correctness via self-refinement, particularly in GPT-4o. The study also highlights challenges in autoformalization accuracy and in evaluation of inherently ambiguous open-ended reasoning tasks.