Is Conversational XAI All You Need? Human-AI Decision Making With a Conversational XAI Assistant

TL;DR

This study investigates how conversational XAI interfaces influence user understanding, trust, and reliance on AI, revealing increased understanding and trust but also highlighting risks of overreliance amplified by LLM-powered conversations.

Contribution

It demonstrates that conversational XAI interfaces improve user understanding and trust compared to dashboards, while also identifying overreliance issues linked to the illusion of explanatory depth.

Findings

Conversational XAI enhances user understanding of AI systems.

Users trust conversational XAI more than dashboards.

Overreliance on AI persists, especially with LLM-powered conversations.

Abstract

Explainable artificial intelligence (XAI) methods are being proposed to help interpret and understand how AI systems reach specific predictions. Inspired by prior work on conversational user interfaces, we argue that augmenting existing XAI methods with conversational user interfaces can increase user engagement and boost user understanding of the AI system. In this paper, we explored the impact of a conversational XAI interface on users' understanding of the AI system, their trust, and reliance on the AI system. In comparison to an XAI dashboard, we found that the conversational XAI interface can bring about a better understanding of the AI system among users and higher user trust. However, users of both the XAI dashboard and conversational XAI interfaces showed clear overreliance on the AI system. Enhanced conversations powered by large language model (LLM) agents amplified over-reliance. Based on our findings, we reason that the potential cause of such overreliance is the illusion of explanatory depth that is concomitant with both XAI interfaces. Our findings have important implications for designing effective conversational XAI interfaces to facilitate appropriate reliance and improve human-AI collaboration. Code can be found at https://github.com/delftcrowd/IUI2025_ConvXAI