Maximal benefits and possible detrimental effects of binary decision aids

TL;DR

This paper analyzes the benefits and potential drawbacks of binary decision aids, showing that poorly integrated aids can sometimes impair user performance and emphasizing careful system design.

Contribution

It introduces a method to compute the maximum benefits of binary aids based on sensitivities and highlights conditions where aids may harm performance.

Findings

Adding aids can yield minimal improvements if detectors are already optimal.

Non-optimal weighting of aids can significantly reduce detection performance.

Introducing users or aids into high-sensitivity systems may decrease overall effectiveness.

Abstract

Binary decision aids, such as alerts, are a simple and widely used form of automation. The formal analysis of a user's task performance with an aid sees the process as the combination of information from two detectors who both receive input about an event and evaluate it. The user's decisions are based on the output of the aid and on the information, the user obtains independently. We present a simple method for computing the maximal benefits a user can derive from a binary aid as a function of the user's and the aid's sensitivities. Combining the user and the aid often adds little to the performance the better detector could achieve alone. Also, if users assign non-optimal weights to the aid, performance may drop dramatically. Thus, the introduction of a valid aid can actually lower detection performance, compared to a more sensitive user working alone. Similarly, adding a user to a system with high sensitivity may lower its performance. System designers need to consider the potential adverse effects of introducing users or aids into systems.