The Role of Explanation Styles and Perceived Accuracy on Decision Making in Predictive Process Monitoring

TL;DR

This study examines how different explanation styles and perceived AI accuracy influence decision-making in predictive process monitoring, highlighting the importance of user-centered evaluation beyond traditional metrics.

Contribution

It introduces an experimental framework to assess the impact of explanation styles and perceived accuracy on user decisions in PPM, emphasizing user trust and decision quality.

Findings

Perceived accuracy significantly affects decision confidence.

Explanation style influences task performance and agreement.

User trust varies with explanation style and perceived accuracy.

Abstract

Predictive Process Monitoring (PPM) often uses deep learning models to predict the future behavior of ongoing processes, such as predicting process outcomes. While these models achieve high accuracy, their lack of interpretability undermines user trust and adoption. Explainable AI (XAI) aims to address this challenge by providing the reasoning behind the predictions. However, current evaluations of XAI in PPM focus primarily on functional metrics (such as fidelity), overlooking user-centered aspects such as their effect on task performance and decision-making. This study investigates the effects of explanation styles (feature importance, rule-based, and counterfactual) and perceived AI accuracy (low or high) on decision-making in PPM. We conducted a decision-making experiment, where users were presented with the AI predictions, perceived accuracy levels, and explanations of different styles. Users' decisions were measured both before and after receiving explanations, allowing the assessment of objective metrics (Task Performance and Agreement) and subjective metrics (Decision Confidence). Our findings show that perceived accuracy and explanation style have a significant effect.