From Stochastic Answers to Verifiable Reasoning: Interpretable Decision-Making with LLM-Generated Code

TL;DR

This paper introduces a method where large language models generate executable, human-readable code for decision-making, improving interpretability, reproducibility, and efficiency in high-stakes tasks like venture capital screening.

Contribution

It reframes LLMs as code generators for deterministic, auditable decision logic, combining statistical validation and iterative refinement without human annotation.

Findings

Achieves 37.5% precision on venture capital screening task

Outperforms GPT-4o in precision while maintaining interpretability

Provides verifiable, executable decision rules over human-readable attributes

Abstract

Large language models (LLMs) are increasingly used for high-stakes decision-making, yet existing approaches struggle to reconcile scalability, interpretability, and reproducibility. Black-box models obscure their reasoning, while recent LLM-based rule systems rely on per-sample evaluation, causing costs to scale with dataset size and introducing stochastic, hallucination-prone outputs. We propose reframing LLMs as code generators rather than per-instance evaluators. A single LLM call generates executable, human-readable decision logic that runs deterministically over structured data, eliminating per-sample LLM queries while enabling reproducible and auditable predictions. We combine code generation with automated statistical validation using precision lift, binomial significance testing, and coverage filtering, and apply cluster-based gap analysis to iteratively refine decision logic without human annotation. We instantiate this framework in venture capital founder screening, a rare-event prediction task with strong interpretability requirements. On VCBench, a benchmark of 4,500 founders with a 9% base success rate, our approach achieves 37.5% precision and an F0.5 score of 25.0%, outperforming GPT-4o (at 30.0% precision and an F0.5 score of 25.7%) while maintaining full interpretability. Each prediction traces to executable rules over human-readable attributes, demonstrating verifiable and interpretable LLM-based decision-making in practice.