Toward Honest Language Models for Deductive Reasoning

TL;DR

This paper investigates how to make language models reason honestly by abstaining when conclusions are not entailed, proposing a reinforcement learning method that improves their ability to do so on graph-based deductive tasks.

Contribution

It introduces ACNCHOR, a reinforcement learning approach that stabilizes training and enhances honest deductive reasoning in language models, addressing limitations of existing methods.

Findings

Prompting and current training methods struggle with honest reasoning.

ACNCHOR stabilizes training and improves reasoning performance.

Ground truth trajectories prevent early training collapse.

Abstract

Deductive reasoning is the process of deriving conclusions strictly from the given premises, without relying on external knowledge. We define honesty in this setting as a model's ability to respond only when the conclusion is logically entailed by the premises, and to abstain otherwise. However, current language models often fail to reason honestly, producing unwarranted answers when the input is insufficient. To study this challenge, we formulate honest deductive reasoning as multi-step tasks where models must either derive the correct conclusion or abstain. We curate two datasets from graph structures, one for linear algebra and one for logical inference, and introduce unanswerable cases by randomly perturbing an edge in half of the instances. We find that prompting and existing training methods, including GRPO with or without supervised fine-tuning initialization, struggle on these tasks. In particular, GRPO optimize only for final task outcomes, leaving models vulnerable to collapse when negative rewards dominate early training. To address this, we propose ACNCHOR, a reinforcement learning method that injects ground truth trajectories into rollouts, preventing early training collapse. Our results demonstrate that this method stabilizes learning and significantly improves the overall reasoning performance, underscoring the importance of training dynamics for enabling honest deductive reasoning in language models.