Make LLMs better zero-shot reasoners: Structure-orientated autonomous reasoning

TL;DR

This paper introduces a structure-oriented analysis method and a multi-agent system called SARA to enhance zero-shot reasoning in LLMs, especially for complex tasks, by improving understanding, guiding reasoning, and reducing errors.

Contribution

The paper proposes a novel structure-oriented analysis approach and a multi-agent reasoning system that significantly improve LLM zero-shot reasoning performance on complex tasks.

Findings

SARA outperforms existing zero-shot methods in complex reasoning tasks.

The structure-oriented analysis improves the understanding and guiding of LLM reasoning.

The system demonstrates robustness against reasoning process attacks.

Abstract

Zero-shot reasoning methods with Large Language Models (LLMs) offer significant advantages including great generalization to novel tasks and reduced dependency on human-crafted examples. However, the current zero-shot methods still have limitations in complex tasks, e.g., answering questions that require multi-step reasoning. In this paper, we address this limitation by introducing a novel structure-oriented analysis method to help LLMs better understand the question and guide the problem-solving process of LLMs. We first demonstrate how the existing reasoning strategies, Chain-of-Thought and ReAct, can benefit from our structure-oriented analysis. In addition to empirical investigations, we leverage the probabilistic graphical model to theoretically explain why our structure-oriented analysis can improve the LLM reasoning process. To further improve the reliability in complex question-answering tasks, we propose a multi-agent reasoning system, Structure-oriented Autonomous Reasoning Agents (SARA), that can better enforce the reasoning process following our structure-oriented analysis by refinement techniques and is equipped with external knowledge retrieval capability to reduce factual errors. Extensive experiments verify the effectiveness of the proposed reasoning system. Surprisingly, in some cases, the system even surpasses few-shot methods. Finally, the system not only improves reasoning accuracy in complex tasks but also demonstrates robustness against potential attacks that corrupt the reasoning process.