Psychophysiological markers of trust in automation: insights from ERP responses in a modified flanker task

TL;DR

This study explores how people develop trust in AI systems by analyzing brain responses during a task involving machine learning reliability, bias, and transparency.

Contribution

The study identifies the late positive component (LPC) as a potential electrophysiological marker of trust in automation.

Findings

LPC amplitude was larger for errors from a more reliable model and correlated with lower subjective trust ratings.

Model transparency had no effect on ERP results or subjective trust ratings.

Direct observation of model performance, rather than textual explanations, influences trust development.

Abstract

This study investigated the sensitivity of event-related potentials (ERP) to factors influencing trust in machine learning (ML) automation, specifically ML reliability, bias, and transparency, with the goal of identifying an electrophysiological marker of trust in automation. Participants performed a flanker task and observed a simulated ML algorithm perform a modified flanker task, while ERP data were collected. The performance flanker task showed canonical patterns in behavioral responses, including fewer errors and shorter response times to congruent trials. We also observed the expected ERP components, including the error-related negativity (ERN) and positivity (Pe), alongside a significant late positive component (LPC) associated with error processing. Contrary to predictions, no differences in oERN amplitudes were observed across model error conditions. The oPe component was elicited by model errors, yet was insensitive to model reliability or bias. Notably, an LPC was also observed to model errors and was larger for errors from the more reliable model (90% vs. 60%). LPC amplitude was negatively correlated with subjective trust ratings in the 60% reliable biased condition, indicating that reduced LPC effects were associated with higher trust levels. These implications of these results are discussed in the context of the P3b and P600 ERP components. Additionally, there were no effects of model transparency on ERP results or subjective trust ratings, suggesting that trust is primarily developed through direct observation of model performance. Our results contribute to understanding the neural mechanisms underlying trust in automation, highlighting the potential of ERP methodologies to advance our understanding in this domain.

This research sheds light on the complex phenomenon of how users develop trust in artificial intelligence (AI) and other automated decision aid systems. Right now, there is widespread inconsistency in how trust is measured across different studies, leading to confusion in the literature about how systems can be designed so that users develop appropriate trust in them. It is important to understand how people learn to trust the output of automated decision aids because they are becoming increasingly prevalent in almost all areas of modern life, and both over-reliance and under-reliance on automated aids could have disastrous consequences. By investigating how explanations of model behavior influence user trust, the study reveals that simply providing textual information about a model’s biases does not necessarily help users better understand its performance. Instead, people tend to rely more on their direct experiences with the model. This finding is crucial as it challenges the common belief that increased transparency in AI outputs will automatically lead to increased appropriate trust. Understanding these dynamics is essential for developing more effective and reliable AI systems that people can trust, especially in high-consequence decision-making situations relevant to national security, like threat detection in overhead imagery or baggage screening, as well as other critical areas of life, like anomaly detection in medical diagnostics. By improving how we understand and measure the development of trust in AI, we can foster better decision-making and ultimately ensure safer and more effective use of this emerging technology.

Figures