Robustness of journal rankings by network flows with different amounts of memory

TL;DR

This paper compares the robustness of different network flow-based journal rankings, showing that higher-order Markov models, which incorporate more memory, are more resilient to journal selection biases but require more data.

Contribution

It introduces a comparison of impact factor, Eigenfactor, and higher-order Markov models for journal ranking robustness, highlighting the advantages of incorporating memory in citation flow models.

Findings

Higher-order Markov models are more robust to journal selection.

Second-order models outperform impact factor and Eigenfactor.

Increased robustness requires more citation data over time.

Abstract

As the number of scientific journals has multiplied, journal rankings have become increasingly important for scientific decisions. From submissions and subscriptions to grants and hirings, researchers, policy makers, and funding agencies make important decisions with influence from journal rankings such as the ISI journal impact factor. Typically, the rankings are derived from the citation network between a selection of journals and unavoidably depend on this selection. However, little is known about how robust rankings are to the selection of included journals. Here we compare the robustness of three journal rankings based on network flows induced on citation networks. They model pathways of researchers navigating scholarly literature, stepping between journals and remembering their previous steps to different degree: zero-step memory as impact factor, one-step memory as Eigenfactor, and two-step memory, corresponding to zero-, first-, and second-order Markov models of citation flow between journals. We conclude that higher-order Markov models perform better and are more robust to the selection of journals. Whereas our analysis indicates that higher-order models perform better, the performance gain for the second-order Markov model comes at the cost of requiring more citation data over a longer time period.