Fine-Tuned Large Language Models for Logical Translation: Reducing Hallucinations with Lang2Logic

TL;DR

This paper presents a framework that combines classical NLP techniques with fine-tuned large language models to translate English statements into formal logic, significantly reducing hallucinations and improving reliability in logical translation tasks.

Contribution

It introduces a novel approach that fine-tunes LLMs with grammar-based methods to minimize hallucinations in translating natural language into formal logic forms.

Findings

Fine-tuned models correct hallucinations effectively.

Reliable CNF generation achieved.

Framework integrates NLP and symbolic computation.

Abstract

Recent advances in natural language processing (NLP), particularly large language models (LLMs), have motivated the automatic translation of natural language statements into formal logic without human intervention. This enables automated reasoning and facilitates debugging, finding loop invariants, and adhering to specifications in software systems. However, hallucinations-incorrect outputs generated by LLMs are challenging, particularly for logical translation tasks requiring precision. This work introduces a novel framework that inputs English sentences, converts them into logical expressions, and then translates them into Conjunctive Normal Form (CNF) for satisfiability solving. It employs classical NLP techniques with self-defined grammar, symbolic computation libraries, and a fine-tuned language model to reduce hallucinations. In the early experiments, we observed that the fine-tuned model, trained on different grammar settings, could intentionally correct the same types of hallucinations made by the original model. Thus, it provides reliable CNF generation.