Recommender systems, stigmergy, and the tyranny of popularity

TL;DR

This paper critiques the dominance of popularity-based algorithms in scientific recommender systems, highlighting their role in reinforcing homogeneity and proposing user-calibrated, autonomy-enhancing platform modifications.

Contribution

It introduces a framework for integrating user-specific calibration and discusses how text embeddings and LLMs can be used to promote diversity and user control in recommender systems.

Findings

Popularity-driven algorithms lead to homogeneity in scientific literature.

User calibration can diversify exposure to research outputs.

Implementing text embeddings and LLMs can enhance user autonomy.

Abstract

Scientific recommender systems, such as Google Scholar and Web of Science, are essential tools for discovery. Search algorithms that power work through stigmergy, a collective intelligence mechanism that surfaces useful paths through repeated engagement. While generally effective, this "rich-get-richer" dynamic results in a small number of high-profile papers that dominate visibility. This essay argues argue that these algorithm over-reliance on popularity fosters intellectual homogeneity and exacerbates structural inequities, stifling innovative and diverse perspectives critical for scientific progress. We propose an overhaul of search platforms to incorporate user-specific calibration, allowing researchers to manually adjust the weights of factors like popularity, recency, and relevance. We also advise platform developers on how text embeddings and LLMs could be implemented in ways that increase user autonomy. While our suggestions are particularly pertinent to aligning recommender systems with scientific values, these ideas are broadly applicable to information access systems in general. Designing platforms that increase user autonomy is an important step toward more robust and dynamic information