Journal Quality Factors from ChatGPT: More meaningful than Impact Factors?

TL;DR

This study explores ChatGPT-generated Journal Quality Factors (JQFs) as potential indicators of journal quality, comparing them with traditional citation metrics and journal rankings across multiple fields, revealing promising correlations.

Contribution

First to estimate academic journal quality using a Large Language Model, demonstrating that JQFs correlate well with journal rankings across diverse research fields.

Findings

JQFs show strong positive correlations with journal ranks in 24 of 25 fields.

Journal citation rates also correlate highly with journal ranks, comparable to JQFs.

Abstract styles may influence JQF results, especially regarding societal context mention.

Abstract

Purpose: Journal Impact Factors and other citation-based indicators are widely used and abused to help select journals to publish in or to estimate the value of a published article. Nevertheless, citation rates primarily reflect scholarly impact rather than other quality dimensions, including societal impact, originality, and rigour. In contrast, Journal Quality Factors (JQFs) are average quality score estimates given to a journal's articles by ChatGPT. Design: JQFs were compared with Polish, Norwegian and Finnish journal ranks and with journal citation rates for 1,300 journals with 130,000 articles from 2021 in large monodisciplinary journals in the 25 out of 27 Scopus broad fields of research for which it was possible. Outliers were also examined. Findings: JQFs correlated positively and mostly strongly (median correlation: 0.641) with journal ranks in 24 out of the 25 broad fields examined, indicating a nearly science-wide ability for ChatGPT to estimate journal quality. Journal citation rates had similarly high correlations with national journal ranks, however, so JQFs are not a universally better indicator. An examination of journals with JQFs not matching their journal ranks suggested that abstract styles may affect the result, such as whether the societal contexts of research are mentioned. Limitations: Different journal rankings may have given different findings because there is no agreed meaning for journal quality. Implications: The results suggest that JQFs are plausible as journal quality indicators in all fields and may be useful for the (few) research and evaluation contexts where journal quality is an acceptable proxy for article quality, and especially for fields like mathematics for which citations are not strong indicators of quality. Originality: This is the first attempt to estimate academic journal value with a Large Language Model.