A Sensitivity Analysis of Pathfinder

TL;DR

This paper conducts a sensitivity analysis of the Pathfinder Bayesian network system to evaluate how parameter precision and structure affect diagnostic performance, aiming to optimize knowledge elicitation efforts.

Contribution

It provides the first detailed sensitivity analysis of Pathfinder, highlighting the impact of parameter noise and structure on system accuracy.

Findings

Parameter noise significantly affects diagnostic accuracy.

Structural changes have a smaller impact than parameter variations.

Sensitivity analysis helps identify where to focus knowledge refinement.

Abstract

Knowledge elicitation is one of the major bottlenecks in expert system design. Systems based on Bayes nets require two types of information--network structure and parameters (or probabilities). Both must be elicited from the domain expert. In general, parameters have greater opacity than structure, and more time is spent in their refinement than in any other phase of elicitation. Thus, it is important to determine the point of diminishing returns, beyond which further refinements will promise little (if any) improvement. Sensitivity analyses address precisely this issue--the sensitivity of a model to the precision of its parameters. In this paper, we report the results of a sensitivity analysis of Pathfinder, a Bayes net based system for diagnosing pathologies of the lymph system. This analysis is intended to shed some light on the relative importance of structure and parameters to system performance, as well as the sensitivity of a system based on a Bayes net to noise in its assessed parameters.