Satisfaction of Assumptions is a Weak Predictor of Performance

TL;DR

This paper introduces a methodology to evaluate the accuracy of uncertain inference systems (UIS) post-optimization, revealing that assumption satisfaction is a weak predictor of their actual performance, which can be comparable or inferior to simple models.

Contribution

It provides a new approach to assess UIS accuracy after parameter optimization, highlighting that assumption correctness does not necessarily correlate with predictive performance.

Findings

UIS's are no more accurate than linear regression on average.

Biased priors can still lead to less accurate UIS's than simple models.

Updating formula importance can outweigh prior assumption correctness.

Abstract

This paper demonstrates a methodology for examining the accuracy of uncertain inference systems (UIS), after their parameters have been optimized, and does so for several common UIS's. This methodology may be used to test the accuracy when either the prior assumptions or updating formulae are not exactly satisfied. Surprisingly, these UIS's were revealed to be no more accurate on the average than a simple linear regression. Moreover, even on prior distributions which were deliberately biased so as give very good accuracy, they were less accurate than the simple probabilistic model which assumes marginal independence between inputs. This demonstrates that the importance of updating formulae can outweigh that of prior assumptions. Thus, when UIS's are judged by their final accuracy after optimization, we get completely different results than when they are judged by whether or not their prior assumptions are perfectly satisfied.