Artificial intelligence technologies to support research assessment: A review

TL;DR

This review explores AI-based indicators and machine learning methods for assessing research quality and impact, including bibliometric analysis, prediction models, and automation tools, with a focus on transparency and bias.

Contribution

It provides a comprehensive overview of AI techniques and indicators used in research assessment, highlighting recent advances and their application in predicting research impact and automating editorial processes.

Findings

Machine learning can predict citation counts and quality scores.

Bibliometric indicators correlate with research impact and quality.

AI tools can automate editorial and review processes.

Abstract

This literature review identifies indicators that associate with higher impact or higher quality research from article text (e.g., titles, abstracts, lengths, cited references and readability) or metadata (e.g., the number of authors, international or domestic collaborations, journal impact factors and authors' h-index). This includes studies that used machine learning techniques to predict citation counts or quality scores for journal articles or conference papers. The literature review also includes evidence about the strength of association between bibliometric indicators and quality score rankings from previous UK Research Assessment Exercises (RAEs) and REFs in different subjects and years and similar evidence from other countries (e.g., Australia and Italy). In support of this, the document also surveys studies that used public datasets of citations, social media indictors or open review texts (e.g., Dimensions, OpenCitations, Altmetric.com and Publons) to help predict the scholarly impact of articles. The results of this part of the literature review were used to inform the experiments using machine learning to predict REF journal article quality scores, as reported in the AI experiments report for this project. The literature review also covers technology to automate editorial processes, to provide quality control for papers and reviewers' suggestions, to match reviewers with articles, and to automatically categorise journal articles into fields. Bias and transparency in technology assisted assessment are also discussed.