Usefulness of altmetrics for measuring the broader impact of research: A case study using data from PLOS (altmetrics) and F1000Prime (paper tags)

TL;DR

This study evaluates the effectiveness of altmetrics, especially social media counts, in measuring the broader societal impact of research papers, using data from F1000Prime and PLOS.

Contribution

It demonstrates that Twitter and Facebook altmetrics are significantly associated with papers tagged as 'good for teaching', indicating broader societal interest.

Findings

Twitter and Facebook counts are higher for 'good for teaching' papers.

Altmetrics from Facebook and Twitter reflect broader societal interest.

Mendeley and Figshare counts do not show this association.

Abstract

Purpose: Whereas citation counts allow the measurement of the impact of research on research itself, an important role in the measurement of the impact of research on other parts of society is ascribed to altmetrics. The present case study investigates the usefulness of altmetrics for measuring the broader impact of research. Methods: This case study is essentially based on a dataset with papers obtained from F1000. The dataset was augmented with altmetrics (such as Twitter counts) which were provided by PLOS (the Public Library of Science). In total, the case study covers a total of 1,082 papers. Findings: The F1000 dataset contains tags on papers which were assigned intellectually by experts and which can characterise a paper. The most interesting tag for altmetric research is "good for teaching". This tag is assigned to papers which could be of interest to a wider circle of readers than the peers in a specialist area. Particularly on Facebook and Twitter, one could expect papers with this tag to be mentioned more often than those without this tag. With respect to the "good for teaching" tag, the results from regression models were able to confirm these expectations: Papers with this tag show significantly higher Facebook and Twitter counts than papers without this tag. This association could not be seen with Mendeley or Figshare counts (that is with counts from platforms which are chiefly of interest in a scientific context). Conclusions: The results of the current study indicate that Facebook and Twitter, but not Figshare or Mendeley, can provide indications of papers which are of interest to a broader circle of readers (and not only for the peers in a specialist area), and seem therefore be useful for societal impact measurement.