Quantitative Methods in Research Evaluation Citation Indicators, Altmetrics, and Artificial Intelligence

TL;DR

This work critically examines the effectiveness of citation metrics, altmetrics, and AI in research evaluation, highlighting their strengths, limitations, and potential applications across various academic fields.

Contribution

It provides a comprehensive analysis of existing indicators and discusses the potential and limitations of AI and large language models in research assessment.

Findings

Citation data can be informative in some fields but are not definitive quality measures.

Altmetrics offer supplementary insights but have limitations in accuracy.

AI, including large language models, may assist in specific research evaluation tasks.

Abstract

This book critically analyses the value of citation data, altmetrics, and artificial intelligence to support the research evaluation of articles, scholars, departments, universities, countries, and funders. It introduces and discusses indicators that can support research evaluation and analyses their strengths and weaknesses as well as the generic strengths and weaknesses of the use of indicators for research assessment. The book includes evidence of the comparative value of citations and altmetrics in all broad academic fields primarily through comparisons against article level human expert judgements from the UK Research Excellence Framework 2021. It also discusses the potential applications of traditional artificial intelligence and large language models for research evaluation, with large scale evidence for the former. The book concludes that citation data can be informative and helpful in some research fields for some research evaluation purposes but that indicators are never accurate enough to be described as research quality measures. It also argues that AI may be helpful in limited circumstances for some types of research evaluation.