Concise and Organized Perception Facilitates Reasoning in Large Language Models

TL;DR

This paper introduces Concise and Organized Perception (COP), a novel method that improves large language models' reasoning by organizing relevant information, leading to significant performance gains on logical and mathematical benchmarks.

Contribution

The paper proposes COP, a new approach that enhances LLM reasoning by systematically organizing input information, addressing brittleness to disorder and distractibility in complex reasoning tasks.

Findings

COP outperforms previous methods on logical benchmarks.

Organized perception improves LLM reasoning accuracy.

Method reduces impact of irrelevant content in reasoning tasks.

Abstract

Exploiting large language models (LLMs) to tackle reasoning has garnered growing attention. It still remains highly challenging to achieve satisfactory results in complex logical problems, characterized by plenty of premises within the context and requiring multi-hop reasoning. In particular, the reasoning capabilities of LLMs are brittle to disorder and distractibility. In this work, we first examine the mechanism from the perspective of information flow and reveal that LLMs confront difficulties akin to human-like cognitive biases when dealing with disordered and irrelevant content in reasoning tasks. However, in contrast to LLMs, disordered and irrelevant content does not significantly decrease human performance, as humans have a propensity to distill the most relevant information and systematically organize their thoughts, aiding them in responding to questions.Stem from that, we further propose a novel reasoning approach named Concise and Organized Perception (COP). COP carefully analyzes the given statements to identify the most pertinent information while eliminating redundancy efficiently. It then prompts the LLMs in a more organized form that adapts to the model's inference process. By perceiving concise and organized context, the reasoning abilities of LLMs can be better elicited. Extensive experimental results on several popular logical benchmarks (ProofWriter, PrOntoQA, PrOntoQA-OOD, and FOLIO) and mathematical benchmark (DI-GSM) show that COP significantly outperforms previous state-of-the-art methods.