Watch Your Steps: Observable and Modular Chains of Thought
Cassandra A. Cohen, William W. Cohen

TL;DR
This paper introduces Program Trace Prompting, a method that formalizes chain of thought explanations using Python syntax to improve observability and enable analysis, achieving strong results across diverse tasks.
Contribution
It presents a novel formalization of chain of thought prompting that enhances observability and analysis, addressing non-local errors and modularity verification.
Findings
Strong performance on 23 BIG-Bench Hard tasks
Identification of non-local errors in reasoning
Methods for verifying step modularity
Abstract
We propose a variant of chain of thought (CoT) prompting called Program Trace Prompting that makes explanations more observable while preserving the power, generality and flexibility of CoT. In our approach, few-shot CoT demonstrations are wrapped in a formal syntax based on Python, and each prompt: identifies and names steps; defines the input/output behavior of steps; and replaces CoT explanations of in-context examples with chains of these formalized steps on the same examples. Program Trace Prompting is applicable to many tasks, achieving strong results on the 23 diverse tasks in the BIG-Bench Hard benchmark. More importantly, by instrumenting explanations in this way, we enable new types of analysis. In particular, we identify "non-local errors" (which correspond to incorrectly learning the reasoning method illustrated in the demonstrations) as an unaddressed issue in CoT learning,…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsCognitive Science and Mapping · Scientific Research and Philosophical Inquiry
