Finding citations for PubMed: A large-scale comparison between five freely available bibliographic data sources

TL;DR

This study compares five free bibliographic data sources for PubMed, evaluating their coverage and citation quality using new metrics and standards, revealing Dimensions as the most comprehensive and accurate source.

Contribution

It introduces a large-scale comparison framework for PubMed citation sources, assessing both coverage and quality with new metrics and standards.

Findings

Dimensions covers 62.4% of PubMed documents, outperforming NIH-OCC.

Over 90% of citations in other sources are also in Dimensions.

Dimensions and NIH-OCC have the best overall citation quality.

Abstract

As an important biomedical database, PubMed provides users with free access to abstracts of its documents. However, citations between these documents need to be collected from external data sources. Although previous studies have investigated the coverage of various data sources, the quality of citations is underexplored. In response, this study compares the coverage and citation quality of five freely available data sources on 30 million PubMed documents, including OpenCitations Index of CrossRef open DOI-to-DOI citations (COCI), Dimensions, Microsoft Academic Graph (MAG), National Institutes of Health Open Citation Collection (NIH-OCC), and Semantic Scholar Open Research Corpus (S2ORC). Three gold standards and five metrics are introduced to evaluate the correctness and completeness of citations. Our results indicate that Dimensions is the most comprehensive data source that provides references for 62.4% of PubMed documents, outperforming the official NIH-OCC dataset (56.7%). Over 90% of citation links in other data sources can also be found in Dimensions. The coverage of MAG, COCI, and S2ORC is 59.6%, 34.7%, and 23.5%, respectively. Regarding the citation quality, Dimensions and NIH-OCC achieve the best overall results. Almost all data sources have a precision higher than 90%, but their recall is much lower. All databases have better performances on recent publications than earlier ones. Meanwhile, the gaps between different data sources have diminished for the documents published in recent years. This study provides evidence for researchers to choose suitable PubMed citation sources, which is also helpful for evaluating the citation quality of free bibliographic databases.