Instance-adaptive Zero-shot Chain-of-Thought Prompting

TL;DR

This paper introduces an instance-adaptive prompting method for zero-shot Chain-of-Thought reasoning in large language models, which dynamically tailors prompts to individual instances, leading to improved reasoning performance across multiple tasks.

Contribution

It proposes a novel instance-adaptive prompting algorithm that analyzes information flow in LLMs to enhance zero-shot reasoning by customizing prompts per instance.

Findings

Consistent performance improvements on math, logic, and commonsense tasks.

Outperforms task-level prompt methods and complex procedures.

Effective across LLaMA-2, LLaMA-3, and Qwen models.

Abstract

Zero-shot Chain-of-Thought (CoT) prompting emerges as a simple and effective strategy for enhancing the performance of large language models (LLMs) in real-world reasoning tasks. Nonetheless, the efficacy of a singular, task-level prompt uniformly applied across the whole of instances is inherently limited since one prompt cannot be a good partner for all, a more appropriate approach should consider the interaction between the prompt and each instance meticulously. This work introduces an instance-adaptive prompting algorithm as an alternative zero-shot CoT reasoning scheme by adaptively differentiating good and bad prompts. Concretely, we first employ analysis on LLMs through the lens of information flow to detect the mechanism under zero-shot CoT reasoning, in which we discover that information flows from question to prompt and question to rationale jointly influence the reasoning results most. We notice that a better zero-shot CoT reasoning needs the prompt to obtain semantic information from the question then the rationale aggregates sufficient information from the question directly and via the prompt indirectly. On the contrary, lacking any of those would probably lead to a bad one. Stem from that, we further propose an instance-adaptive prompting strategy (IAP) for zero-shot CoT reasoning. Experiments conducted with LLaMA-2, LLaMA-3, and Qwen on math, logic, and commonsense reasoning tasks (e.g., GSM8K, MMLU, Causal Judgement) obtain consistent improvement, demonstrating that the instance-adaptive zero-shot CoT prompting performs better than other task-level methods with some curated prompts or sophisticated procedures, showing the significance of our findings in the zero-shot CoT reasoning mechanism.