Prompt Architecture Determines Reasoning Quality: A Variable Isolation Study on the Car Wash Problem

TL;DR

This study shows that the way prompts are structured, especially using the STAR framework, significantly improves reasoning accuracy in language models on the car wash problem, with structured scaffolds outperforming context injection.

Contribution

It demonstrates that prompt architecture, particularly the STAR framework, is crucial for reasoning performance, surpassing context injection methods in implicit physical constraint inference.

Findings

STAR framework raises accuracy from 0% to 85%.

Adding user profile context improves accuracy by 10%.

Full-stack approach achieves 100% accuracy.

Abstract

Large language models consistently fail the "car wash problem," a viral reasoning benchmark requiring implicit physical constraint inference. We present a variable isolation study (n=20 per condition, 6 conditions, 120 total trials) examining which prompt architecture layers in a production system enable correct reasoning. Using Claude 3.5 Sonnet with controlled hyperparameters (temperature 0.7, top_p 1.0), we find that the STAR (Situation-Task-Action-Result) reasoning framework alone raises accuracy from 0% to 85% (p=0.001, Fisher's exact test, odds ratio 13.22). Adding user profile context via vector database retrieval provides a further 10 percentage point gain, while RAG context contributes an additional 5 percentage points, achieving 100% accuracy in the full-stack condition. These results suggest that structured reasoning scaffolds -- specifically, forced goal articulation before inference -- matter substantially more than context injection for implicit constraint reasoning tasks.