Logic.py: Bridging the Gap between LLMs and Constraint Solvers

TL;DR

This paper introduces Logic.py, a method that combines large language models with constraint solvers by formalising problems in a domain-specific language, significantly improving performance on logic puzzles.

Contribution

It proposes a novel framework that formalises search problems in a logic-focused DSL and solves them with constraint solvers, enhancing LLM capabilities.

Findings

Achieved over 90% accuracy on ZebraLogicBench

65% absolute improvement over baseline Llama 3.1 70B

Demonstrated effective hybrid approach combining LLMs and solvers

Abstract

We present a novel approach to formalise and solve search-based problems using large language models, which significantly improves upon previous state-of-the-art results. We demonstrate the efficacy of this approach on the logic puzzles benchmark ZebraLogicBench. Instead of letting the LLM attempt to directly solve the puzzles, our method prompts the model to formalise the problem in a logic-focused domain-specific language (DSL) called Logic.py. This formalised representation is then solved using a constraint solver, leveraging the strengths of both the language model and the solver. Our approach achieves a remarkable 65% absolute improvement over the baseline performance of Llama 3.1 70B on ZebraLogicBench, setting a new state-of-the-art with an accuracy of over 90%. This significant advancement demonstrates the potential of combining language models with domain-specific languages and auxiliary tools on traditionally challenging tasks for LLMs.