Scalable Delphi: Large Language Models for Structured Risk Estimation

TL;DR

This paper introduces Scalable Delphi, a method using large language models to perform structured risk estimation efficiently, matching expert judgments and reducing assessment time from months to minutes.

Contribution

It adapts the Delphi method for LLMs with diverse personas and iterative refinement, enabling scalable and calibrated risk assessments in high-stakes domains.

Findings

LLM panels achieve high correlation with ground truth (r=0.87-0.95)

Elicitation time reduced from months to minutes

LLM-based elicitation aligns well with human expert panels

Abstract

Quantitative risk assessment in high-stakes domains relies on structured expert elicitation to estimate unobservable properties. The gold standard - the Delphi method - produces calibrated, auditable judgments but requires months of coordination and specialist time, placing rigorous risk assessment out of reach for most applications. We investigate whether Large Language Models (LLMs) can serve as scalable proxies for structured expert elicitation. We propose Scalable Delphi, adapting the classical protocol for LLMs with diverse expert personas, iterative refinement, and rationale sharing. Because target quantities are typically unobservable, we develop an evaluation framework based on necessary conditions: calibration against verifiable proxies, sensitivity to evidence, and alignment with human expert judgment. We evaluate in the domain of AI-augmented cybersecurity risk, using three capability benchmarks and independent human elicitation studies. LLM panels achieve strong correlations with benchmark ground truth (Pearson r=0.87-0.95), improve systematically as evidence is added, and align with human expert panels - in one comparison, closer to a human panel than the two human panels are to each other. This demonstrates that LLM-based elicitation can extend structured expert judgment to settings where traditional methods are infeasible, reducing elicitation time from months to minutes.