Improving Human Decisions by Adjusting the Alerting Thresholds for Computer Alerting Tools According to User and Task Characteristics

TL;DR

Adjusting alerting thresholds in computer alerting tools based on user ability and task difficulty can significantly improve decision accuracy, highlighting the importance of personalized settings for optimal performance.

Contribution

This study demonstrates that tailoring alert thresholds to individual user ability and task difficulty enhances decision accuracy, challenging the assumption of a single optimal threshold.

Findings

Higher ability users benefit from more sensitive thresholds on easier tasks.

Adjusting thresholds can reduce decision errors by up to 30%.

User preferences did not align with thresholds that improved performance.

Abstract

Objective: To investigate whether performance (number of correct decisions) of humans supported by a computer alerting tool can be improved by tailoring the tool's alerting threshold (sensitivity/specificity combination) according to user ability and task difficulty. Background: Many researchers implicitly assume that for each tool there exists a single ideal threshold. But research shows the effects of alerting tools on decision errors to vary depending on variables such as user ability and task difficulty. These findings motivated our investigation. Method: Forty-seven participants edited text passages, aided by a simulated spell-checker tool. We experimentally manipulated passage difficulty and tool alerting threshold, measured participants' editing and dictation ability, and counted participants' decision errors (false positives + false negatives). We investigated whether alerting threshold, user ability, task difficulty and their interactions affected error count. Results: Which alerting threshold better helped a user depended on an interaction between user ability and passage difficulty. Some effects were large: for higher ability users, a more sensitive threshold reduced errors by 30%, on the easier passages. Participants were not significantly more likely to prefer the alerting threshold with which they performed better. Conclusion: Adjusting alerting thresholds for individual users' ability and task difficulty could substantially improve effects of automated alerts on user decisions. This potential deserves further investigation. Improvement size and rules for adjustment will be application-specific. Application: Computer alerting tools have critical roles in many domains. Designers should assess potential benefits of adjustable alerting thresholds for their specific CAT application. Guidance for choosing thresholds will be essential for realizing these benefits in practice.