Break the Chain: Large Language Models Can be Shortcut Reasoners

TL;DR

This paper introduces 'break the chain' strategies that enable large language models to use shortcuts for reasoning, reducing complexity and improving efficiency in complex logical and commonsense tasks.

Contribution

It proposes novel zero-shot prompting methods and a new dataset, ShortcutQA, to evaluate shortcut reasoning in language models, advancing beyond traditional chain-of-thought approaches.

Findings

Models effectively use shortcuts with 'break the chain' strategies

ShortcutQA challenges models' reasoning abilities

Shortcuts improve reasoning speed and efficiency

Abstract

Recent advancements in Chain-of-Thought (CoT) reasoning utilize complex modules but are hampered by high token consumption, limited applicability, and challenges in reproducibility. This paper conducts a critical evaluation of CoT prompting, extending beyond arithmetic to include complex logical and commonsense reasoning tasks, areas where standard CoT methods fall short. We propose the integration of human-like heuristics and shortcuts into language models (LMs) through "break the chain" strategies. These strategies disrupt traditional CoT processes using controlled variables to assess their efficacy. Additionally, we develop innovative zero-shot prompting strategies that encourage the use of shortcuts, enabling LMs to quickly exploit reasoning clues and bypass detailed procedural steps. Our comprehensive experiments across various LMs, both commercial and open-source, reveal that LMs maintain effective performance with "break the chain" strategies. We also introduce ShortcutQA, a dataset specifically designed to evaluate reasoning through shortcuts, compiled from competitive tests optimized for heuristic reasoning tasks such as forward/backward reasoning and simplification. Our analysis confirms that ShortcutQA not only poses a robust challenge to LMs but also serves as an essential benchmark for enhancing reasoning efficiency in AI.