Generative AI and the future of scientometrics: current topics and future questions

TL;DR

This paper reviews the use of Generative AI in scientometrics, discussing its capabilities, limitations, and potential impact on measuring scientific knowledge, while emphasizing the need for systematic evaluation and theoretical reflection.

Contribution

It provides a critical analysis of GenAI applications in scientometrics and highlights the importance of empirical comparison and theoretical insights for future research.

Findings

GenAI excels in language generation tasks like topic labeling.

GenAI faces limitations in tasks requiring stable semantics or structured knowledge.

Performance of GenAI models should be systematically compared for specific tasks.

Abstract

The aim of this paper is to review the use of GenAI in scientometrics, and to begin a debate on the broader implications for the field. First, we provide an introduction on GenAI's generative and probabilistic nature as rooted in distributional linguistics. And we relate this to the debate on the extent to which GenAI might be able to mimic human 'reasoning'. Second, we leverage this distinction for a critical engagement with recent experiments using GenAI in scientometrics, including topic labelling, the analysis of citation contexts, predictive applications, scholars' profiling, and research assessment. GenAI shows promise in tasks where language generation dominates, such as labelling, but faces limitations in tasks that require stable semantics, pragmatic reasoning, or structured domain knowledge. However, these results might become quickly outdated. Our recommendation is, therefore, to always strive to systematically compare the performance of different GenAI models for specific tasks. Third, we inquire whether, by generating large amounts of scientific language, GenAI might have a fundamental impact on our field by affecting textual characteristics used to measure science, such as authors, words, and references. We argue that careful empirical work and theoretical reflection will be essential to remain capable of interpreting the evolving patterns of knowledge production.