A Framework for Comparing Uncertain Inference Systems to Probability

TL;DR

This paper introduces a framework to compare various uncertain inference systems (UISs), assessing their assumptions and performance in propagating uncertainty, using the maximum entropy principle as a standard benchmark.

Contribution

It presents a novel experimental framework for comparing UISs, highlighting how different systems make assumptions about correlations and their impact on inference accuracy.

Findings

Different UISs vary significantly in their assumptions and performance.

The maximum entropy principle provides a useful benchmark for comparison.

Experimental results illustrate the differences among UISs in practice.

Abstract

Several different uncertain inference systems (UISs) have been developed for representing uncertainty in rule-based expert systems. Some of these, such as Mycin's Certainty Factors, Prospector, and Bayes' Networks were designed as approximations to probability, and others, such as Fuzzy Set Theory and DempsterShafer Belief Functions were not. How different are these UISs in practice, and does it matter which you use? When combining and propagating uncertain information, each UIS must, at least by implication, make certain assumptions about correlations not explicily specified. The maximum entropy principle with minimum cross-entropy updating, provides a way of making assumptions about the missing specification that minimizes the additional information assumed, and thus offers a standard against which the other UISs can be compared. We describe a framework for the experimental comparison of the performance of different UISs, and provide some illustrative results.