Microsoft Academic Automatic Document Searches: Accuracy for Journal Articles and Suitability for Citation Analysis

TL;DR

This study evaluates the accuracy of Microsoft Academic's search methods for journal articles and its suitability for citation analysis, finding high correlation with Scopus counts but noting some indexing biases.

Contribution

It identifies effective search strategies for retrieving journal articles in Microsoft Academic and assesses their citation count reliability compared to Scopus.

Findings

90% of journal articles found using title and DOI filtering

89% of articles found using title filtering without DOI

Microsoft Academic citation counts strongly correlate with Scopus counts (average 0.95)

Abstract

Microsoft Academic is a free academic search engine and citation index that is similar to Google Scholar but can be automatically queried. Its data is potentially useful for bibliometric analysis if it is possible to search effectively for individual journal articles. This article compares different methods to find journal articles in its index by searching for a combination of title, authors, publication year and journal name and uses the results for the widest published correlation analysis of Microsoft Academic citation counts for journal articles so far. Based on 126,312 articles from 323 Scopus subfields in 2012, the optimal strategy to find articles with DOIs is to search for them by title and filter out those with incorrect DOIs. This finds 90% of journal articles. For articles without DOIs, the optimal strategy is to search for them by title and then filter out matches with dissimilar metadata. This finds 89% of journal articles, with an additional 1% incorrect matches. The remaining articles seem to be mainly not indexed by Microsoft Academic or indexed with a different language version of their title. From the matches, Scopus citation counts and Microsoft Academic counts have an average Spearman correlation of 0.95, with the lowest for any single field being 0.63. Thus, Microsoft Academic citation counts are almost universally equivalent to Scopus citation counts for articles that are not recent but there are national biases in the results.