Towards More Accountable Search Engines: Online Evaluation of Representation Bias

TL;DR

This paper proposes a method to measure societal representation bias in search engine results, specifically evaluating gender bias in Google's Knowledge Graph, to promote accountability and informed user judgment.

Contribution

It introduces a novel approach for online bias evaluation in search engines, focusing on societal features like gender representation.

Findings

Identified gender bias in Google's Knowledge Graph Carousel for occupations

Developed a measurement framework for societal bias in search results

Demonstrated the applicability of the method to real-world search engine data

Abstract

Information availability affects people's behavior and perception of the world. Notably, people rely on search engines to satisfy their need for information. Search engines deliver results relevant to user requests usually without being or making themselves accountable for the information they deliver, which may harm people's lives and, in turn, society. This potential risk urges the development of evaluation mechanisms of bias in order to empower the user in judging the results of search engines. In this paper, we give a possible solution to measuring representation bias with respect to societal features for search engines and apply it to evaluating the gender representation bias for Google's Knowledge Graph Carousel for listing occupations.