Are Language Models Efficient Reasoners? A Perspective from Logic Programming

TL;DR

This paper evaluates the reasoning efficiency of language models by comparing their generated proofs to shortest logical proofs, revealing that models often make unnecessary inferences and struggle with irrelevant information.

Contribution

It introduces a novel framework for assessing LM reasoning efficiency using logic programming and constructs a dataset with irrelevant information to test models.

Findings

Current LMs' accuracy declines with irrelevant information.

Generated proofs often include unnecessary inferences.

Models struggle to ignore distractions in reasoning tasks.

Abstract

Modern language models (LMs) exhibit strong deductive reasoning capabilities, yet standard evaluations emphasize correctness while overlooking a key aspect of reasoning: efficiency. In real-world reasoning scenarios, much of the available information is irrelevant, and effective deductive inference requires identifying and ignoring such distractions. We propose a framework for assessing LM reasoning efficiency through the lens of logic programming, introducing a simple method to align proofs written in natural language -- as generated by an LM -- with shortest proofs found by executing the logic program. Efficiency is quantified by measuring how well a model avoids unnecessary inference. Empirically, we construct a dataset of math word problems injected with various number of irrelevant axioms that vary in semantic overlap with the goal theorem. We find that current LMs show marked accuracy declines under such conditions -- even with minimal, domain-consistent distractions -- and the proofs they generate frequently exhibit detours through irrelevant inferences.