Emerging Reliance Behaviors in Human-AI Content Grounded Data Generation: The Role of Cognitive Forcing Functions and Hallucinations

TL;DR

This study examines how hallucinations and Cognitive Forcing Functions influence user reliance and data quality in human-AI content generation, revealing that hallucinations impair quality and overreliance can occur even with mitigation strategies.

Contribution

It provides empirical insights into reliance behaviors and the effects of hallucinations and Cognitive Forcing Functions in human-LLM collaborative data creation.

Findings

Hallucinations significantly reduce data quality.

Cognitive Forcing Functions influence user integration of AI responses.

Overreliance on AI leads to lower quality data.

Abstract

We investigate the impact of hallucinations and Cognitive Forcing Functions in human-AI collaborative content-grounded data generation, focusing on the use of Large Language Models (LLMs) to assist in generating high quality conversational data. Through a study with 34 users who each completed 8 tasks (n=272), we found that hallucinations significantly reduce data quality. While Cognitive Forcing Functions do not always alleviate these effects, their presence influences how users integrate AI responses. Specifically, we observed emerging reliance behaviors, with users often appending AI-generated responses to their correct answers, even when the AI's suggestions conflicted. This points to a potential drawback of Cognitive Forcing Functions, particularly when AI suggestions are inaccurate. Users who overrelied on AI-generated text produced lower quality data, emphasizing the nuanced dynamics of overreliance in human-LLM collaboration compared to traditional human-AI decision-making.