Text and Patterns: For Effective Chain of Thought, It Takes Two to Tango
Aman Madaan, Amir Yazdanbakhsh

TL;DR
This paper investigates the mechanisms behind chain of thought prompting in large language models, revealing that factual patterns are less critical than previously thought and emphasizing the symbiotic role of text and patterns in effective few-shot learning.
Contribution
The study systematically analyzes prompt components using counterfactual prompts across multiple models, challenging conventional wisdom and elucidating the roles of text and patterns in CoT prompting.
Findings
Factual patterns in prompts are largely immaterial to CoT success.
Intermediate steps serve as signals for symbol replication rather than learning facilitation.
Text and patterns work together, with text providing commonsense and patterns guiding task understanding.
Abstract
The past decade has witnessed dramatic gains in natural language processing and an unprecedented scaling of large language models. These developments have been accelerated by the advent of few-shot techniques such as chain of thought (CoT) prompting. Specifically, CoT pushes the performance of large language models in a few-shot setup by augmenting the prompts with intermediate steps. Despite impressive results across various tasks, the reasons behind their success have not been explored. This work uses counterfactual prompting to develop a deeper understanding of CoT-based few-shot prompting mechanisms in large language models. We first systematically identify and define the key components of a prompt: symbols, patterns, and text. Then, we devise and conduct an exhaustive set of experiments across four different tasks, by querying the model with counterfactual prompts where only one of…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsTopic Modeling · Natural Language Processing Techniques · Software Engineering Research
MethodsRefunds@Expedia|||How do I get a full refund from Expedia? · 15 Ways to Contact How can i speak to someone at Delta Airlines · Attention Is All You Need · Linear Layer · Cosine Annealing · {Dispute@FaQ-s}How to file a dispute with Expedia? · Multi-Head Attention · Softmax · Linear Warmup With Cosine Annealing · Attention Dropout
