Can Google Scholar and Mendeley help to assess the scholarly impacts of dissertations?

TL;DR

This study develops a multistage method to extract Google Scholar citation counts for dissertations, compares them with Mendeley reader counts, and assesses their usefulness for impact evaluation across disciplines and age groups.

Contribution

Introduces a new multistage extraction method for Google Scholar citations of dissertations and compares its effectiveness with Mendeley reader counts for impact assessment.

Findings

Google Scholar citations are more useful for older dissertations, especially in social sciences and humanities.

Mendeley reader counts are more effective for recent dissertations less than two years old.

The method achieves over 95% precision in extracting citation data from Google Scholar.

Abstract

Dissertations can be the single most important scholarly outputs of junior researchers. Whilst sets of journal articles are often evaluated with the help of citation counts from the Web of Science or Scopus, these do not index dissertations and so their impact is hard to assess. In response, this article introduces a new multistage method to extract Google Scholar citation counts for large collections of dissertations from repositories indexed by Google. The method was used to extract Google Scholar citation counts for 77,884 American doctoral dissertations from 2013-2017 via ProQuest, with a precision of over 95%. Some ProQuest dissertations that were dual indexed with other repositories could not be retrieved with ProQuest-specific searches but could be found with Google Scholar searches of the other repositories. The Google Scholar citation counts were then compared with Mendeley reader counts, a known source of scholarly-like impact data. A fifth of the dissertations had at least one citation recorded in Google Scholar and slightly fewer had at least one Mendeley reader. Based on numerical comparisons, the Mendeley reader counts seem to be more useful for impact assessment purposes for dissertations that are less than two years old, whilst Google Scholar citations are more useful for older dissertations, especially in social sciences, arts and humanities. Google Scholar citation counts may reflect a more scholarly type of impact than that of Mendeley reader counts because dissertations attract a substantial minority of their citations from other dissertations. In summary, the new method now makes it possible for research funders, institutions and others to systematically evaluate the impact of dissertations, although additional Google Scholar queries for other online repositories are needed to ensure comprehensive coverage.