Validity of altmetrics data for measuring societal impact: A study using data from Altmetric and F1000Prime

TL;DR

This study evaluates whether altmetric data can validly measure societal impact by analyzing a large dataset, finding that altmetrics correlate with societal relevance and suggesting topic-based normalization for better assessment.

Contribution

It provides empirical evidence on the validity of altmetrics for societal impact measurement and proposes a topic-based normalization approach.

Findings

Papers tagged 'good for teaching' have higher altmetric counts.

Scientifically oriented 'new finding' papers show higher citation counts.

Normalization of altmetric data should be based on topics, not broad subject categories.

Abstract

Can altmetric data be validly used for the measurement of societal impact? The current study seeks to answer this question with a comprehensive dataset (about 100,000 records) from very disparate sources (F1000, Altmetric, and an in-house database based on Web of Science). In the F1000 peer review system, experts attach particular tags to scientific papers which indicate whether a paper could be of interest for science or rather for other segments of society. The results show that papers with the tag "good for teaching" do achieve higher altmetric counts than papers without this tag - if the quality of the papers is controlled. At the same time, a higher citation count is shown especially by papers with a tag that is specifically scientifically oriented ("new finding"). The findings indicate that papers tailored for a readership outside the area of research should lead to societal impact. If altmetric data is to be used for the measurement of societal impact, the question arises of its normalization. In bibliometrics, citations are normalized for the papers' subject area and publication year. This study has taken a second analytic step involving a possible normalization of altmetric data. As the results show there are particular scientific topics which are of especial interest for a wide audience. Since these more or less interesting topics are not completely reflected in Thomson Reuters' journal sets, a normalization of altmetric data should not be based on the level of subject categories, but on the level of topics.