Understanding the Effect of Algorithm Transparency of Model Explanations in Text-to-SQL Semantic Parsing

TL;DR

This study explores how varying levels of algorithm transparency in text-to-SQL models influence user reliance, engagement, and trust, revealing medium transparency as optimal for balanced user experience and improved performance.

Contribution

It introduces three levels of model explanation for text-to-SQL parsing and empirically evaluates their effects on user trust, reliance, and performance.

Findings

Medium transparency explanations balance user reliance and engagement.

High transparency reduces user reliance, low transparency increases reliance.

Only medium transparency users improved performance over time.

Abstract

Explaining the decisions of AI has become vital for fostering appropriate user trust in these systems. This paper investigates explanations for a structured prediction task called ``text-to-SQL Semantic Parsing'', which translates a natural language question into a structured query language (SQL) program. In this task setting, we designed three levels of model explanation, each exposing a different amount of the model's decision-making details (called ``algorithm transparency''), and investigated how different model explanations could potentially yield different impacts on the user experience. Our study with $\sim$100 participants shows that (1) the low-/high-transparency explanations often lead to less/more user reliance on the model decisions, whereas the medium-transparency explanations strike a good balance. We also show that (2) only the medium-transparency participant group was able to engage further in the interaction and exhibit increasing performance over time, and that (3) they showed the least changes in trust before and after the study.