Learning When Not to Decide: A Framework for Overcoming Factual Presumptuousness in AI Adjudication

TL;DR

This paper addresses AI presumptuousness in legal decision-making by developing a framework that improves accuracy and appropriately defers decisions when evidence is incomplete, enhancing reliability in high-stakes settings.

Contribution

It introduces SPEC, a structured prompting framework that significantly reduces presumptuousness and improves decision accuracy in legal AI adjudication.

Findings

Standard AI approaches achieve only 15% accuracy with insufficient information.

Advanced prompting improves accuracy but often over-corrects, leading to unnecessary deferrals.

SPEC achieves 89% overall accuracy and appropriately defers when evidence is lacking.

Abstract

A well-known limitation of AI systems is presumptuousness: the tendency of AI systems to provide confident answers when information may be lacking. This challenge is particularly acute in legal applications, where a core task for attorneys, judges, and administrators is to determine whether evidence is sufficient to reach a conclusion. We study this problem in the important setting of unemployment insurance adjudication, which has seen rapid integration of AI systems and where the question of additional fact-finding poses the most significant bottleneck for a system that affects millions of applicants annually. First, through a collaboration with the Colorado Department of Labor and Employment, we secure rare access to official training materials and guidance to design a novel benchmark that systematically varies in information completeness. Second, we evaluate four leading AI platforms and show that standard RAG-based approaches achieve an average of only 15% accuracy when information is insufficient. Third, advanced prompting methods improve accuracy on inconclusive cases but over-correct, withholding decisions even on clear cases. Fourth, we introduce a structured framework requiring explicit identification of missing information before any determination (SPEC, Structured Prompting for Evidence Checklists). SPEC achieves 89% overall accuracy, while appropriately deferring when evidence is insufficient -- demonstrating that presumptuousness in legal AI is systematic but addressable, and that doing so is a necessary step towards systems that reliably support, rather than supplant, human judgment wherever decisions must await sufficient evidence.