All Eyes on the Ranker: Participatory Auditing to Surface Blind Spots in Ranked Search Results

TL;DR

This study demonstrates that participatory auditing involving users can reveal impacts, accountability gaps, and limitations of search ranking systems, especially as models become more sophisticated and convincing.

Contribution

The paper introduces a participatory auditing approach with workshops that uncover user-perceived impacts and limitations of search rankings, highlighting the importance of user involvement in accountability.

Findings

Participatory auditing reveals user-perceived impacts across multiple dimensions.

Neural models' perceived competence can reduce critical scrutiny during audits.

Participants desire greater visibility into the full search pipeline and recourse options.

Abstract

Search engines that present users with a ranked list of search results are a fundamental technology for providing public access to information. Evaluations of such systems are typically conducted by domain experts and focus on model-centric metrics, relevance judgments, or output-based analyses, rather than on how accountability, harm, or trust are experienced by users. This paper argues that participatory auditing is essential for revealing users' causal and contextual understandings of how ranked search results produce impacts, particularly as ranking models appear increasingly convincing and sophisticated in their semantic interpretation of user queries. We report on three participatory auditing workshops (n=21) in which participants engaged with a custom search interface across four tasks, comparing a lexical ranker (BM25) and a neural semantic reranker (MonoT5), exploring varying levels of transparency and user controls, and examining an intentionally adversarially manipulated ranking. Reflexive activities prompted participants to articulate causal narratives linking search system properties to broader impacts. Synthesising the findings, we contribute a taxonomy of user-perceived impacts of ranked search results, spanning epistemic, representational, infrastructural, and downstream social impacts. However, interactions with the neural model revealed limits to participatory auditing itself: perceived system competence and accumulated trust reduced critical scrutiny during the workshop, allowing manipulations to go undetected. Participants expressed desire for visibility into the full search pipeline and recourse mechanisms. Together, these findings show how participatory auditing can surface user perceived impacts and accountability gaps that remain unseen when relying on conventional audits, while revealing where participatory auditing may encounter limitations.