Enhancing Logical Reasoning in Language Models via Symbolically-Guided Monte Carlo Process Supervision

TL;DR

This paper introduces a novel method combining symbolic reasoning trajectories with Monte Carlo estimation and preference optimization to enhance logical reasoning and generalization in large language models.

Contribution

It proposes a scalable approach to synthesize symbolic reasoning trajectories and train models to improve reasoning reliability and out-of-domain generalization.

Findings

Significant performance improvements on FOLIO and LogicAsker benchmarks.

Enhanced out-of-domain generalization in claim verification tasks.

Effective leveraging of symbolic trajectories via Monte Carlo estimation.

Abstract

Large language models (LLMs) have shown strong performance in many reasoning benchmarks. However, recent studies have pointed to memorization, rather than generalization, as one of the leading causes for such performance. LLMs, in fact, are susceptible to content variations, demonstrating a lack of robust planning or symbolic abstractions supporting their reasoning process. To improve reliability, many attempts have been made to combine LLMs with symbolic methods. Nevertheless, existing approaches fail to effectively leverage symbolic representations due to the challenges involved in developing reliable and scalable verification mechanisms. In this paper, we propose to overcome such limitations by synthesizing high-quality symbolic reasoning trajectories with stepwise pseudo-labels at scale via Monte Carlo estimation. A Process Reward Model (PRM) can be efficiently trained based on the synthesized data and then used to select more symbolic trajectories. The trajectories are then employed with Direct Preference Optimization (DPO) and Supervised Fine-Tuning (SFT) to improve logical reasoning and generalization. Our results on benchmarks (i.e., FOLIO and LogicAsker) show the effectiveness of the proposed method with gains on frontier and open-weight models. Moreover, additional experiments on claim verification data reveal that fine-tuning on the generated symbolic reasoning trajectories enhances out-of-domain generalizability, suggesting the potential impact of the proposed method in enhancing planning and logical reasoning.