A Knowledge Engineer's Comparison of Three Evidence Aggregation Methods

TL;DR

This paper compares three evidence aggregation methods used by knowledge engineers, emphasizing practical factors like error impact on system performance over purely theoretical correctness.

Contribution

It introduces a practical evaluation of evidence aggregation methods considering real-world error effects, beyond traditional probabilistic accuracy.

Findings

Error impact varies among methods

Some methods outperform others in practical scenarios

Theoretical correctness does not guarantee best real-world performance

Abstract

The comparisons of uncertainty calculi from the last two Uncertainty Workshops have all used theoretical probabilistic accuracy as the sole metric. While mathematical correctness is important, there are other factors which should be considered when developing reasoning systems. These other factors include, among other things, the error in uncertainty measures obtainable for the problem and the effect of this error on the performance of the resulting system.