Assessment, Criticism and Improvement of Imprecise Subjective Probabilities for a Medical Expert System

TL;DR

This study evaluates the accuracy and reliability of subjective probability assessments made by pediatric cardiologists for a belief network on congenital heart disease, using data from 200 babies and Brier scoring.

Contribution

It introduces a method to assess and improve imprecise subjective probabilities in expert systems through empirical validation and adaptation with observed data.

Findings

Assessments are generally reliable but tend to be overly extreme.

Imprecise judgments can be converted into implicit samples for analysis.

Probabilities adapt with experience when combined with observed data.

Abstract

Three paediatric cardiologists assessed nearly 1000 imprecise subjective conditional probabilities for a simple belief network representing congenital heart disease, and the quality of the assessments has been measured using prospective data on 200 babies. Quality has been assessed by a Brier scoring rule, which decomposes into terms measuring lack of discrimination and reliability. The results are displayed for each of 27 diseases and 24 questions, and generally the assessments are reliable although there was a tendency for the probabilities to be too extreme. The imprecision allows the judgements to be converted to implicit samples, and by combining with the observed data the probabilities naturally adapt with experience. This appears to be a practical procedure even for reasonably large expert systems.