Who Gets Recommended? Investigating Gender, Race, and Country Disparities in Paper Recommendations from Large Language Models

TL;DR

This study evaluates large language models' literature recommendation capabilities, revealing biases towards highly cited, recent, and large-team papers, but no evidence of gender, race, or country biases in recommendations.

Contribution

The paper provides a comprehensive analysis of biases in LLM-based literature recommendations, highlighting their tendencies and contrasting them with human biases.

Findings

LLMs favor highly cited, recent, and large-team papers

No evidence of gender, race, or country biases in recommendations

Recommendation accuracy remains limited

Abstract

This paper investigates the performance of several representative large models in the tasks of literature recommendation and explores potential biases in research exposure. The results indicate that not only LLMs' overall recommendation accuracy remains limited but also the models tend to recommend literature with greater citation counts, later publication date, and larger author teams. Yet, in scholar recommendation tasks, there is no evidence that LLMs disproportionately recommend male, white, or developed-country authors, contrasting with patterns of known human biases.