IDEA: An Interpretable and Editable Decision-Making Framework for LLMs via Verbal-to-Numeric Calibration

TL;DR

IDEA is a novel framework that makes large language model decision-making interpretable, editable, and well-calibrated by extracting decision knowledge into a parametric model with human-AI collaboration capabilities.

Contribution

It introduces a method to extract and calibrate LLM decision knowledge into an interpretable, editable model with mathematical guarantees, enhancing high-stakes decision-making.

Findings

IDEA outperforms existing models like DeepSeek R1 and GPT-5.2 in accuracy.

Achieves perfect factor exclusion and exact calibration.

Enables precise human-AI collaboration through direct parameter editing.

Abstract

Large Language Models are increasingly deployed for decision-making, yet their adoption in high-stakes domains remains limited by miscalibrated probabilities, unfaithful explanations, and inability to incorporate expert knowledge precisely. We propose IDEA, a framework that extracts LLM decision knowledge into an interpretable parametric model over semantically meaningful factors. Through joint learning of verbal-to-numerical mappings and decision parameters via EM, correlated sampling that preserves factor dependencies, and direct parameter editing with mathematical guarantees, IDEA produces calibrated probabilities while enabling quantitative human-AI collaboration. Experiments across five datasets show IDEA with Qwen-3-32B (78.6%) outperforms DeepSeek R1 (68.1%) and GPT-5.2 (77.9%), achieving perfect factor exclusion and exact calibration -- precision unattainable through prompting alone. The implementation is publicly available at https://github.com/leonbig/IDEA.