LLM-AR: LLM-powered Automated Reasoning Framework

TL;DR

This paper introduces LLM-AR, a novel framework that combines large language models with probabilistic reasoning to improve decision accuracy and interpretability in high-stakes applications like startup success prediction.

Contribution

The paper presents LLM-AR, a neural-symbolic pipeline that distills LLM heuristics into probabilistic rules and iteratively refines them, enhancing reliability and interpretability.

Findings

Achieves 59.5% precision and 8.7% recall on unseen data

Outperforms random baseline by 5.9x in precision

Provides interpretable decision paths for human review

Abstract

Large language models (LLMs) can already identify patterns and reason effectively, yet their variable accuracy hampers adoption in high-stakes decision-making applications. In this paper, we study this issue from a venture capital perspective by predicting idea-stage startup success based on founder traits. (i) To build a reliable prediction model, we introduce LLM-AR, a pipeline inspired by neural-symbolic systems that distils LLM-generated heuristics into probabilistic rules executed by the ProbLog automated-reasoning engine. (ii) An iterative policy-evolution loop incorporates association-rule mining to progressively refine the prediction rules. On unseen folds, LLM-AR achieves 59.5% precision and 8.7% recall, 5.9x the random baseline precision, while exposing every decision path for human inspection. The framework is interpretable and tunable via hyperparameters, showing promise to extend into other domains.