The Unique Taste of LLMs for Papers: Potential issues in Using LLMs for Digital Library Document Recommendation Tasks

TL;DR

This paper evaluates large language models' effectiveness in literature recommendation tasks, highlighting moderate accuracy, bias considerations, and their tendency to recommend timely and collaborative research without amplifying inequalities.

Contribution

It provides an empirical assessment of LLMs in literature recommendation, revealing their strengths, limitations, and bias tendencies in scholarly contexts.

Findings

Models offer somewhat satisfactory recommendations after manual screening.

Overall recommendation accuracy remains moderate.

No evidence of bias amplification related to gender, race, or country development.

Abstract

This paper investigates the performance of several representative large models in the field of literature recommendation and explores potential biases. The results indicate that while some large models' recommendations can be somewhat satisfactory after simple manual screening, overall, the accuracy of these models in specific literature recommendation tasks is generally moderate. Additionally, the models tend to recommend literature that is timely, collaborative, and expands or deepens the field. In scholar recommendation tasks. There is no evidence to suggest that LLMs exacerbate inequalities related to gender, race, or the level of development of countries.