Interaction Configurations and Prompt Guidance in Conversational AI for Question Answering in Human-AI Teams

TL;DR

This study explores how different prompt guidance configurations in conversational AI can improve human-AI collaboration in question answering, revealing that nudging strategies enhance response quality more effectively than traditional methods.

Contribution

The paper introduces and empirically evaluates two novel prompt guidance configurations—Nudging and Highlight—for improving human-AI collaboration in question answering tasks.

Findings

Nudging improves answer quality compared to AI alone.

Effective collaboration depends on the configuration used.

Combining human and AI efforts does not always lead to better outcomes.

Abstract

Understanding the dynamics of human-AI interaction in question answering is crucial for enhancing collaborative efficiency. Extending from our initial formative study, which revealed challenges in human utilization of conversational AI support, we designed two configurations for prompt guidance: a Nudging approach, where the AI suggests potential responses for human agents, and a Highlight strategy, emphasizing crucial parts of reference documents to aid human responses. Through two controlled experiments, the first involving 31 participants and the second involving 106 participants, we compared these configurations against traditional human-only approaches, both with and without AI assistance. Our findings suggest that effective human-AI collaboration can enhance response quality, though merely combining human and AI efforts does not ensure improved outcomes. In particular, the Nudging configuration was shown to help improve the quality of the output when compared to AI alone. This paper delves into the development of these prompt guidance paradigms, offering insights for refining human-AI collaborations in conversational question-answering contexts and contributing to a broader understanding of human perceptions and expectations in AI partnerships.