Semi-structured LLM Reasoners Can Be Rigorously Audited

TL;DR

This paper introduces Semi-Structured Reasoning Models (SSRMs) that produce audit-friendly reasoning traces, enabling automatic detection of errors without sacrificing model accuracy across multiple benchmarks.

Contribution

The paper presents SSRMs that generate semi-structured reasoning traces, facilitating rigorous auditing methods to detect reasoning errors in large language models.

Findings

SSRMs produce audit-friendly reasoning traces.

Auditing methods effectively identify reasoning errors.

SSRMs maintain high accuracy across benchmarks.

Abstract

Although Large Language Models (LLMs) have become capable reasoners, the problem of faithfulness persists: their reasoning can contain errors and omissions that are difficult to detect and that may obscure biases in model outputs. To address this issue, we introduce Semi-Structured Reasoning Models (SSRMs), which are trained to produce semi-structured representations of reasoning. SSRMs generate reasoning traces in a non-executable Pythonic syntax that names each reasoning step and marks its inputs and outputs. This structure allows SSRM traces to be automatically audited to identify reasoning flaws. We evaluate three types of audits: hand-crafted structured reasoning audits, written in a domain-specific language (DSL) implemented in Python; LLM-generated structured reasoning audits; and learned typicality audits, which apply probabilistic models over reasoning traces. We show that all of these methods can be used to effectively flag probable reasoning errors. Importantly, the auditability of SSRMs does not appear to compromise overall accuracy: in evaluation on twelve benchmarks and two model families, SSRMs demonstrate strong performance and generalizability relative to other models of comparable size.