Evidence for studying interactions between science and policy: An exploration of scholarly and policy references in Overton-indexed policy documents

TL;DR

This study analyzes a large dataset of policy documents from Overton to understand the presence and distribution of scholarly and policy references, revealing coverage biases and assessing Overton's potential for policy impact analysis.

Contribution

It provides the first large-scale systematic analysis of reference data in Overton, highlighting its potential and limitations for policy metrics and evidence-based policymaking.

Findings

7.7% of documents contain scholarly references

10.6% contain policy references

Significant coverage biases across dimensions

Abstract

Overton, a global policy index, provides new opportunities to study the interactions between science and policy. This study aims to characterize the presence of scholarly and policy references in Overton-indexed policy documents and examine their distribution across key bibliographic dimensions, thereby assessing Overton's potential as a data source for policy metrics. We analyze a dataset of approximately 17.5 million policy documents from Overton, incorporating metadata such as publication year, policy source, country, language, subject area, and policy topic. Descriptive statistics are employed to assess the presence and distribution of reference data across these dimensions. Overton indexes a substantial volume of policy documents and identifies considerable reference data within them: 7.7% of documents contain scholarly references and 10.6% contain policy references. However, the presence of references varies significantly across publications years, source types, countries, languages, subject areas, and policy topics, indicating coverage biases that may affect interpretations of policy impact. The analysis is based on the Overton database as of June 2025. As Overton is regularly updated, the distribution patterns of indexed documents and references may evolve over time. The findings offer insights into the opportunities and constraints of using Overton for investigating evidence-based policymaking and for assessing the policy uptake of research outputs in the context of research evaluation. This is the first large-scale study to systematically examine the distribution of reference data in Overton. It contributes a foundational understanding of this emerging source for policy metrics, highlighting both its potential applications and limitations, and underlining the importance of addressing current coverage imbalances.