Evidential Reasoning in a Network Usage Prediction Testbed

TL;DR

This empirical study compares evidential reasoning techniques in predicting network states, finding all outperform humans and highlighting how different metrics influence method performance, with no clear best among the tested calculi.

Contribution

The paper introduces a methodology for comparing evidential reasoning techniques in network prediction, focusing on how performance varies with different evaluation metrics.

Findings

All calculi outperformed human subjects.

Performance depends on the chosen metric: net favors point-valued methods, yield favors interval-valued methods.

No clear winner among the tested evidential reasoning methods.

Abstract

This paper reports on empirical work aimed at comparing evidential reasoning techniques. While there is prima facie evidence for some conclusions, this i6 work in progress; the present focus is methodology, with the goal that subsequent results be meaningful. The domain is a network of UNIX* cycle servers, and the task is to predict properties of the state of the network from partial descriptions of the state. Actual data from the network are taken and used for blindfold testing in a betting game that allows abstention. The focal technique has been Kyburg's method for reasoning with data of varying relevance to a particular query, though the aim is to be able eventually to compare various uncertainty calculi. The conclusions are not novel, but are instructive. 1. All of the calculi performed better than human subjects, so unbiased access to sample experience is apparently of value. 2. Performance depends on metric: (a) when trials are repeated, net = gains - losses favors methods that place many bets, if the probability of placing a correct bet is sufficiently high; that is, it favors point-valued formalisms; (b) yield = gains/(gains + lossee) favors methods that bet only when sure to bet correctly; that is, it favors interval-valued formalisms. 3. Among the calculi, there were no clear winners or losers. Methods are identified for eliminating the bias of the net as a performance criterion and for separating the calculi effectively: in both cases by posting odds for the betting game in the appropriate way.