Navigating the Thin Line: Examining User Behavior in Search to Detect Engagement and Backfire Effects

TL;DR

This study investigates how different levels of bias and AI stance labels in search results influence opinionated users' content diversity consumption and search behaviors on debated topics.

Contribution

It provides empirical evidence on how biased search results and stance labels affect opinionated users' engagement and stance diversity during web searches.

Findings

Counter-attitudinal bias increases stance-opposing content consumption.

Bias correlates with fewer interactions on search pages.

Stance labels enhance stance diversity, especially with biased results.

Abstract

Opinionated users often seek information that aligns with their preexisting beliefs while dismissing contradictory evidence due to confirmation bias. This conduct hinders their ability to consider alternative stances when searching the web. Despite this, few studies have analyzed how the diversification of search results on disputed topics influences the search behavior of highly opinionated users. To this end, we present a preregistered user study (n = 257) investigating whether different levels (low and high) of bias metrics and search results presentation (with or without AI-predicted stances labels) can affect the stance diversity consumption and search behavior of opinionated users on three debated topics (i.e., atheism, intellectual property rights, and school uniforms). Our results show that exposing participants to (counter-attitudinally) biased search results increases their consumption of attitude-opposing content, but we also found that bias was associated with a trend toward overall fewer interactions within the search page. We also found that 19% of users interacted with queries and search pages but did not select any search results. When we removed these participants in a post-hoc analysis, we found that stance labels increased the diversity of stances consumed by users, particularly when the search results were biased. Our findings highlight the need for future research to explore distinct search scenario settings to gain insight into opinionated users' behavior.