Logic Sketch Prompting (LSP): A Deterministic and Interpretable Prompting Method

TL;DR

Logic Sketch Prompting (LSP) is a deterministic, interpretable framework for large language models that significantly improves accuracy and consistency in rule-based tasks, making it suitable for safety-critical applications.

Contribution

LSP introduces typed variables, deterministic evaluators, and rule-based validation to enhance LLM reasoning with traceability and repeatability.

Findings

LSP outperforms zero shot, concise, and chain of thought prompting in accuracy and F1 scores.

LSP achieves 0.83 to 0.89 accuracy and F1 across tasks and models.

Statistically significant improvements are confirmed by McNemar tests.

Abstract

Large language models (LLMs) excel at natural language reasoning but remain unreliable on tasks requiring strict rule adherence, determinism, and auditability. Logic Sketch Prompting (LSP) is a lightweight prompting framework that introduces typed variables, deterministic condition evaluators, and a rule based validator that produces traceable and repeatable outputs. Using two pharmacologic logic compliance tasks, we benchmark LSP against zero shot prompting, chain of thought prompting, and concise prompting across three open weight models: Gemma 2, Mistral, and Llama 3. Across both tasks and all models, LSP consistently achieves the highest accuracy (0.83 to 0.89) and F1 score (0.83 to 0.89), substantially outperforming zero shot prompting (0.24 to 0.60), concise prompts (0.16 to 0.30), and chain of thought prompting (0.56 to 0.75). McNemar tests show statistically significant gains for LSP across nearly all comparisons (p < 0.01). These results demonstrate that LSP improves determinism, interpretability, and consistency without sacrificing performance, supporting its use in clinical, regulated, and safety critical decision support systems.