Investigating Bias in Political Search Query Suggestions by Relative Comparison with LLMs

TL;DR

This paper presents a novel method combining large language models, pairwise comparison, and Elo scoring to detect and quantify bias in political search query suggestions, revealing differences between Google and Bing.

Contribution

It introduces a new approach for bias detection in search suggestions using LLMs and Elo scoring, specifically applied to political queries in the U.S. news domain.

Findings

Bias detection in search suggestions is feasible with the proposed method.

Differences in bias levels are observed between Google and Bing.

The approach effectively quantifies bias in politically sensitive search suggestions.

Abstract

Search query suggestions affect users' interactions with search engines, which then influences the information they encounter. Thus, bias in search query suggestions can lead to exposure to biased search results and can impact opinion formation. This is especially critical in the political domain. Detecting and quantifying bias in web search engines is difficult due to its topic dependency, complexity, and subjectivity. The lack of context and phrasality of query suggestions emphasizes this problem. In a multi-step approach, we combine the benefits of large language models, pairwise comparison, and Elo-based scoring to identify and quantify bias in English search query suggestions. We apply our approach to the U.S. political news domain and compare bias in Google and Bing.