BIRD: A Trustworthy Bayesian Inference Framework for Large Language Models

TL;DR

BIRD is a new probabilistic inference framework that leverages large language models to produce more accurate and trustworthy probability estimates, improving decision-making in real-world tasks.

Contribution

BIRD introduces a novel method combining Bayesian networks with LLM abductions to enhance probabilistic estimation accuracy.

Findings

BIRD achieves 30% better probability estimation accuracy than LLM baselines.

The framework improves trustworthiness of decision-making processes.

It effectively aligns Bayesian networks with LLM-generated abductive factors.

Abstract

Predictive models often need to work with incomplete information in real-world tasks. Consequently, they must provide reliable probability or confidence estimation, especially in large-scale decision-making and planning tasks. Current large language models (LLMs) are insufficient for accurate estimations, but they can generate relevant factors that may affect the probabilities, produce coarse-grained probabilities when the information is more complete, and help determine which factors are relevant to specific downstream contexts. In this paper, we make use of these capabilities of LLMs to provide a significantly more accurate probabilistic estimation. We propose BIRD, a novel probabilistic inference framework that aligns a Bayesian network with LLM abductions and then estimates more accurate probabilities in a deduction step. We show BIRD provides reliable probability estimations that are 30% better than those provided directly by LLM baselines. These estimates further contribute to better and more trustworthy decision making.