LOGIC-LM++: Multi-Step Refinement for Symbolic Formulations

TL;DR

Logic-LM++ enhances formal reasoning in large language models by employing multi-step refinement and pairwise comparison, significantly improving accuracy on complex reasoning benchmarks.

Contribution

It introduces Logic-LM++, a novel refinement method that improves formal language generation in LLMs through pairwise evaluation, outperforming previous models.

Findings

Achieves 18.5% improvement on standard prompting

Outperforms Logic-LM on three datasets

Demonstrates effectiveness of multi-step refinement

Abstract

In this paper we examine the limitations of Large Language Models (LLMs) for complex reasoning tasks. Although recent works have started to employ formal languages as an intermediate representation for reasoning tasks, they often face challenges in accurately generating and refining these formal specifications to ensure correctness. To address these issues, this paper proposes Logic-LM++, an improvement on Logic-LM . It uses the ability of LLMs to do pairwise comparisons, allowing the evaluation of the refinements suggested by the LLM. The paper demonstrates that Logic-LM++ outperforms Logic-LM and other contemporary techniques across natural language reasoning tasks on three datasets, FOLIO, ProofWriter and AR-LSAT, with an average improvement of 18.5% on standard prompting, 12.3% on chain of thought prompting and 5% on Logic-LM.