A Comparison of Decision Analysis and Expert Rules for Sequential Diagnosis

TL;DR

This paper compares decision theoretic methods and heuristic rules for motorcycle engine diagnosis, finding that decision analysis reduces expected diagnostic time by 14% over expert rules, with robust results despite estimate inaccuracies.

Contribution

It provides an experimental comparison showing decision analytic algorithms outperform heuristic expert rules in fault diagnosis efficiency.

Findings

Decision analysis reduces expected diagnosis time by 14%.

Results are robust to inaccuracies in probability and cost estimates.

Supports the use of decision theoretic methods over heuristic rules.

Abstract

There has long been debate about the relative merits of decision theoretic methods and heuristic rule-based approaches for reasoning under uncertainty. We report an experimental comparison of the performance of the two approaches to troubleshooting, specifically to test selection for fault diagnosis. We use as experimental testbed the problem of diagnosing motorcycle engines. The first approach employs heuristic test selection rules obtained from expert mechanics. We compare it with the optimal decision analytic algorithm for test selection which employs estimated component failure probabilities and test costs. The decision analytic algorithm was found to reduce the expected cost (i.e. time) to arrive at a diagnosis by an average of 14% relative to the expert rules. Sensitivity analysis shows the results are quite robust to inaccuracy in the probability and cost estimates. This difference suggests some interesting implications for knowledge acquisition.