On the performativity of SDG classifications in large bibliometric databases

TL;DR

This paper investigates how SDG classifications in major bibliometric databases influence scientific visibility and impact measurement, highlighting biases and the implications of using large language models to analyze these effects.

Contribution

It introduces a method using fine-tuned large language models to analyze data biases introduced by diverse SDG classifications in bibliometric data.

Findings

High sensitivity of LLMs to model architecture and fine-tuning processes

Wide arbitrariness in SDG classifications raises research concerns

Potential biases affect the reliability of impact assessments

Abstract

Large bibliometric databases, such as Web of Science, Scopus, and OpenAlex, facilitate bibliometric analyses, but are performative, affecting the visibility of scientific outputs and the impact measurement of participating entities. Recently, these databases have taken up the UN's Sustainable Development Goals (SDGs) in their respective classifications, which have been criticised for their diverging nature. This work proposes using the feature of large language models (LLMs) to learn about the "data bias" injected by diverse SDG classifications into bibliometric data by exploring five SDGs. We build a LLM that is fine-tuned in parallel by the diverse SDG classifications inscribed into the databases' SDG classifications. Our results show high sensitivity in model architecture, classified publications, fine-tuning process, and natural language generation. The wide arbitrariness at different levels raises concerns about using LLM in research practice.