Extracting Probabilistic Knowledge from Large Language Models for Bayesian Network Parameterization

TL;DR

This paper investigates using Large Language Models to extract probabilistic knowledge for Bayesian Network parameterization, demonstrating their potential to generate meaningful priors across various domains and improve modeling with limited data.

Contribution

It introduces a novel method for leveraging LLMs to derive probabilistic priors for Bayesian Networks and establishes the first comprehensive baseline for evaluating LLMs in this context.

Findings

LLMs can produce meaningful probabilistic estimates for BN parameters.

Querying LLMs outperforms random and uniform baselines.

LLM-derived priors enhance Bayesian modeling, especially with scarce data.

Abstract

In this work, we evaluate the potential of Large Language Models (LLMs) in building Bayesian Networks (BNs) by approximating domain expert priors. LLMs have demonstrated potential as factual knowledge bases; however, their capability to generate probabilistic knowledge about real-world events remains understudied. We explore utilizing the probabilistic knowledge inherent in LLMs to derive probability estimates for statements regarding events and their relationships within a BN. Using LLMs in this context allows for the parameterization of BNs, enabling probabilistic modeling within specific domains. Our experiments on eighty publicly available Bayesian Networks, from healthcare to finance, demonstrate that querying LLMs about the conditional probabilities of events provides meaningful results when compared to baselines, including random and uniform distributions, as well as approaches based on next-token generation probabilities. We explore how these LLM-derived distributions can serve as expert priors to refine distributions extracted from data, especially when data is scarce. Overall, this work introduces a promising strategy for automatically constructing Bayesian Networks by combining probabilistic knowledge extracted from LLMs with real-world data. Additionally, we establish the first comprehensive baseline for assessing LLM performance in extracting probabilistic knowledge.