The Impact of AI Generated Content on Decision Making for Topics Requiring Expertise

TL;DR

This study investigates how AI-generated content influences online decision-making, especially on expert-level topics, revealing that users rely heavily on provided information and that AI content is as helpful as human sources, raising both potential and concerns.

Contribution

It provides empirical evidence on the influence of AI-generated content on decision-making for domain-specific topics requiring expertise, highlighting user reliance and perceptions.

Findings

Participants less likely to change opinions on expert topics.

AI-generated and human-written information are equally helpful.

Users rely on provided conclusions due to difficulty in independent reasoning.

Abstract

Modelling users' online decision-making and opinion change is a complex issue that needs to consider users' personal determinants, the nature of the topic and the information retrieval activities. Furthermore, generative-AIbased products like ChatGPT gradually become an essential element for the retrieval of online information. However, the interaction between domainspecific knowledge and AI-generated content during online decision-making is unclear. We conducted a lab-based explanatory sequential study with university students to overcome this research gap. In the experiment, we surveyed participants about a set of general domain topics that are easy to grasp and another set of domain-specific topics that require adequate levels of chemical science knowledge to fully comprehend. We provided participants with decision-supporting information that was either produced using generative AI or collected from selected expert human-written sources to explore the role of AI-generated content compared to ordinary information during decision-making. Our result revealed that participants are less likely to change opinions on domain-specific topics. Since participants without professional knowledge had difficulty performing in-depth and independent reasoning based on the information, they favoured relying on conclusions presented in the provided materials and tended to stick to their initial opinion. Besides, information that is labelled as AI-generated is equivalently helpful as information labelled as dedicatedly human-written for participants in this experiment, indicating the vast potential as well as concerns for AI replacing human experts to help users tackle professional topics or issues.