Hypothesis Testing Prompting Improves Deductive Reasoning in Large Language Models

TL;DR

This paper introduces Hypothesis Testing Prompting, a novel method that enhances deductive reasoning in large language models by incorporating assumptions, backward reasoning, and fact verification, leading to more accurate and logical outputs.

Contribution

The paper presents a new prompting technique that improves deductive reasoning accuracy and reasoning process quality in large language models.

Findings

Significant improvement in reasoning accuracy on ProofWriter and RuleTaker datasets.

Generation of more reasonable and standardized reasoning processes.

Enhanced ability to verify conclusions during reasoning steps.

Abstract

Combining different forms of prompts with pre-trained large language models has yielded remarkable results on reasoning tasks (e.g. Chain-of-Thought prompting). However, along with testing on more complex reasoning, these methods also expose problems such as invalid reasoning and fictional reasoning paths. In this paper, we develop \textit{Hypothesis Testing Prompting}, which adds conclusion assumptions, backward reasoning, and fact verification during intermediate reasoning steps. \textit{Hypothesis Testing prompting} involves multiple assumptions and reverses validation of conclusions leading to its unique correct answer. Experiments on two challenging deductive reasoning datasets ProofWriter and RuleTaker show that hypothesis testing prompting not only significantly improves the effect, but also generates a more reasonable and standardized reasoning process.