Policy documents as sources for measuring societal impact: How often is climate change research mentioned in policy-related documents?

TL;DR

This study explores the potential of using Altmetric data on policy document mentions to measure the societal impact of climate change research, revealing low coverage but highlighting certain influential publication types.

Contribution

It provides an initial empirical analysis of policy document mentions in climate change research, demonstrating the limited but valuable use of Altmetric data for impact assessment.

Findings

Only 1.2% of climate change publications are mentioned in policy documents.

Publications in Nature and Science are more frequently cited in policy.

Research in Earth sciences and social geography is more relevant in policy contexts.

Abstract

In the current UK Research Excellence Framework (REF) and the Excellence in Research for Australia (ERA) societal impact measurements are inherent parts of the national evaluation systems. In this study, we deal with a relatively new form of societal impact measurements. Recently, Altmetric - a start-up providing publication level metrics - started to make data for publications available which have been mentioned in policy documents. We regard this data source as an interesting possibility to specifically measure the (societal) impact of research. Using a comprehensive dataset with publications on climate change as an example, we study the usefulness of the new data source for impact measurement. Only 1.2% (n=2,341) out of 191,276 publications on climate change in the dataset have at least one policy mention. We further reveal that papers published in Nature and Science as well as from the areas "Earth and related environmental sciences" and "Social and economic geography" are especially relevant in the policy context. Given the low coverage of the climate change literature in policy documents, this study can be only a first attempt to study this new source of altmetric data. Further empirical studies are necessary in upcoming years, because mentions in policy documents are of special interest in the use of altmetric data for measuring target-oriented the broader impact of research.