Unveiling the Magic of Code Reasoning through Hypothesis Decomposition and Amendment

TL;DR

This paper introduces a novel code reasoning benchmark and a reflective hypothesis decomposition pipeline to improve large language models' reasoning capabilities, achieving significant performance gains and better handling of complex, real-world tasks.

Contribution

The paper proposes a new code reasoning benchmark and a human-inspired hypothesis decomposition pipeline that enhances reasoning accuracy and robustness in large language models.

Findings

LLMs struggle with identifying satisfactory reasoning pathways.

The RHDA pipeline improves reasoning performance up to 3x.

Applying the pipeline to real-world tasks enhances failure handling.

Abstract

The reasoning abilities are one of the most enigmatic and captivating aspects of large language models (LLMs). Numerous studies are dedicated to exploring and expanding the boundaries of this reasoning capability. However, tasks that embody both reasoning and recall characteristics are often overlooked. In this paper, we introduce such a novel task, code reasoning, to provide a new perspective for the reasoning abilities of LLMs. We summarize three meta-benchmarks based on established forms of logical reasoning, and instantiate these into eight specific benchmark tasks. Our testing on these benchmarks reveals that LLMs continue to struggle with identifying satisfactory reasoning pathways. Additionally, we present a new pathway exploration pipeline inspired by human intricate problem-solving methods. This Reflective Hypothesis Decomposition and Amendment (RHDA) pipeline consists of the following iterative steps: (1) Proposing potential hypotheses based on observations and decomposing them; (2) Utilizing tools to validate hypotheses and reflection outcomes; (3) Revising hypothesis in light of observations. Our approach effectively mitigates logical chain collapses arising from forgetting or hallucination issues in multi-step reasoning, resulting in performance gains of up to $3\times$. Finally, we expanded this pipeline by applying it to simulate complex household tasks in real-world scenarios, specifically in VirtualHome, enhancing the handling of failure cases. We release our code and all of results at https://github.com/TnTWoW/code_reasoning.