In which fields do higher impact journals publish higher quality articles?

TL;DR

This study investigates the relationship between journal impact indicators and expert judgments of article quality across various scientific fields, finding only weak to moderate correlations and questioning the reliability of impact metrics as quality proxies.

Contribution

It provides a comprehensive analysis of the correlation between journal impact factors and expert-rated article quality across all scientific fields, highlighting the limitations of impact metrics.

Findings

Weak to moderate correlation between impact and quality across fields

Highest correlation was 0.54 in Infectious Diseases

Impact indicators are never fully reliable for assessing article quality

Abstract

The Journal Impact Factor and other indicators that assess the average citation rate of articles in a journal are consulted by many academics and research evaluators, despite initiatives against overreliance on them. Despite this, there is limited evidence about the extent to which journal impact indicators in any field relates to human judgements about the journals or their articles. In response, we compared average citation rates of journals against expert judgements of their articles in all fields of science. We used preliminary quality scores for 96,031 articles published 2014-18 from the UK Research Excellence Framework (REF) 2021. We show that whilst there is a positive correlation between expert judgements of article quality and average journal impact in all fields of science, it is very weak in many fields and is never strong. The strength of the correlation varies from 0.11 to 0.43 for the 27 broad fields of Scopus. The highest correlation for the 94 Scopus narrow fields with at least 750 articles was only 0.54, for Infectious Diseases, and there was only one negative correlation, for the mixed category Computer Science (all). The results suggest that the average citation impact of a Scopus-indexed journal is never completely irrelevant to the quality of an article, even though it is never a strong indicator of article quality.