Evaluating Active Learning Heuristics for Sequential Diagnosis

TL;DR

This paper evaluates various active learning heuristics for sequential diagnosis to determine the most effective strategies for identifying system failures efficiently in real-world cases.

Contribution

It provides an empirical comparison of multiple heuristics for measurement selection in sequential diagnosis, highlighting their relative strengths and situational effectiveness.

Findings

Some heuristics outperform others in specific scenarios

Performance depends on system complexity and fault characteristics

Guidelines for heuristic selection based on diagnosis context

Abstract

Given a malfunctioning system, sequential diagnosis aims at identifying the root cause of the failure in terms of abnormally behaving system components. As initial system observations usually do not suffice to deterministically pin down just one explanation of the system's misbehavior, additional system measurements can help to differentiate between possible explanations. The goal is to restrict the space of explanations until there is only one (highly probable) explanation left. To achieve this with a minimal-cost set of measurements, various (active learning) heuristics for selecting the best next measurement have been proposed. We report preliminary results of extensive ongoing experiments with a set of selection heuristics on real-world diagnosis cases. In particular, we try to answer questions such as "Is some heuristic always superior to all others?", "On which factors does the (relative) performance of the particular heuristics depend?" or "Under which circumstances should I use which heuristic?"