CRISP: Complex Reasoning with Interpretable Step-based Plans

TL;DR

CRISP introduces a multi-domain dataset of high-level plans for complex reasoning tasks, demonstrating that fine-tuning models on this data enhances plan quality and reasoning performance across various domains.

Contribution

The paper presents CRISP, a new dataset of automatically generated, validated high-level plans, and shows that fine-tuning models on CRISP improves reasoning and plan quality beyond few-shot prompting.

Findings

Fine-tuning on CRISP yields higher-quality plans than larger models with few-shot prompting.

Models fine-tuned on CRISP outperform Chain-of-Thought reasoning.

Cross-domain training on CRISP improves plan generation in different domains.

Abstract

Recent advancements in large language models (LLMs) underscore the need for stronger reasoning capabilities to solve complex problems effectively. While Chain-of-Thought (CoT) reasoning has been a step forward, it remains insufficient for many domains. A promising alternative is explicit high-level plan generation, but existing approaches largely assume that LLMs can produce effective plans through few-shot prompting alone, without additional training. In this work, we challenge this assumption and introduce CRISP (Complex Reasoning with Interpretable Step-based Plans), a multi-domain dataset of high-level plans for mathematical reasoning and code generation. The plans in CRISP are automatically generated and rigorously validated--both intrinsically, using an LLM as a judge, and extrinsically, by evaluating their impact on downstream task performance. We demonstrate that fine-tuning a small model on CRISP enables it to generate higher-quality plans than much larger models using few-shot prompting, while significantly outperforming Chain-of-Thought reasoning. Furthermore, our out-of-domain evaluation reveals that fine-tuning on one domain improves plan generation in the other, highlighting the generalizability of learned planning capabilities.