The Effects of Research Level and Article Type on the Differences between Citation Metrics and F1000 Recommendations

TL;DR

This study explores how research level and article type influence the differences between citation metrics and F1000 recommendations, highlighting that article type significantly affects these differences and suggesting a balanced evaluation approach.

Contribution

It provides new insights into how article type impacts citation and recommendation discrepancies, informing more nuanced research evaluation methods.

Findings

Non-primary research is more highly cited than recommended.

Transformative research is more recommended but less cited.

Article type significantly influences citation and recommendation differences.

Abstract

F1000 recommendations have been validated as a potential data source for research evaluation, but reasons for differences between F1000 Article Factor (FFa scores) and citations remain to be explored. By linking 28254 publications in F1000 to citations in Scopus, we investigated the effect of research level and article type on the internal consistency of assessments based on citations and FFa scores. It turns out that research level has little impact, while article type has big effect on the differences. These two measures are significantly different for two groups: non-primary research or evidence-based research publications are more highly cited rather than highly recommended, however, translational research or transformative research publications are more highly recommended by faculty members but gather relatively lower citations. This can be expected because citation activities are usually practiced by academic authors while the potential for scientific revolutions and the suitability for clinical practice of an article should be investigated from the practitioners' points of view. We conclude with a policy relevant recommendation that the application of bibliometric approaches in research evaluation procedures should include the proportion of three types of publications: evidence-based research, transformative research, and translational research. The latter two types are more suitable to be assessed through peer review.