What Else Do I Need to Know? The Effect of Background Information on Users' Reliance on QA Systems

TL;DR

This paper investigates how background information influences users' reliance on QA system predictions, revealing that while it helps detect errors, it also increases confidence in both correct and incorrect answers.

Contribution

The study demonstrates the dual effect of background information on user reliance and confidence, highlighting challenges in supporting user verification of QA outputs.

Findings

Background info reduces over-reliance on incorrect answers.

Providing background increases user confidence in judgments.

Users often rely on predictions even without sufficient info to verify.

Abstract

NLP systems have shown impressive performance at answering questions by retrieving relevant context. However, with the increasingly large models, it is impossible and often undesirable to constrain models' knowledge or reasoning to only the retrieved context. This leads to a mismatch between the information that the models access to derive the answer and the information that is available to the user to assess the model predicted answer. In this work, we study how users interact with QA systems in the absence of sufficient information to assess their predictions. Further, we ask whether adding the requisite background helps mitigate users' over-reliance on predictions. Our study reveals that users rely on model predictions even in the absence of sufficient information needed to assess the model's correctness. Providing the relevant background, however, helps users better catch model errors, reducing over-reliance on incorrect predictions. On the flip side, background information also increases users' confidence in their accurate as well as inaccurate judgments. Our work highlights that supporting users' verification of QA predictions is an important, yet challenging, problem.