An analysis of the effects of open science indicators on citations in the French Open Science Monitor

TL;DR

This study examines how open science practices like pre-prints, data sharing, and software sharing correlate with increased citation counts in French publications from 2020 to 2022, highlighting discipline-specific variations.

Contribution

It provides empirical evidence of positive correlations between open science indicators and citations, with detailed analysis across disciplines and open access status.

Findings

Pre-prints are associated with a 19% increase in citations.

Software sharing correlates with a 13.5% increase in citations.

Data sharing is linked to a 14.3% increase in citations.

Abstract

This study investigates the correlation of citation impact with various open science indicators (OSI) within the French Open Science Monitor (FOSM), a dataset comprising approximately 900,000 publications authored by French authors from 2020 to 2022. By integrating data from OpenAlex and Crossref, we analyze open science indicators such as the presence of a pre-print, data sharing, and software sharing in 576,537 publications in the FOSM dataset. Our analysis reveals a positive correlation between these OSI and citation counts. Considering our most complete citation prediction model, we find pre-prints are correlated with a significant positive effect of 19% on citation counts, software sharing of 13.5%, and data sharing of 14.3%. We find large variations in the correlations of OSIs with citations in different research disciplines, and observe that open access status of publications is correlated with a 8.6% increase in citations in our model. While these results remain observational and are limited to the scope of the analysis, they suggest a consistent correlation between citation advantages and open science indicators. Our results may be valuable to policy makers, funding agencies, researchers, publishers, institutions, and other stakeholders who are interested in understanding the academic impacts, or effects, of open science practices.

Tables

Table 1. Table 1: Publication counts by BSO category.

BSO category	Publication counts
Medical research	160’672
Biology (fond.)	95’019
Earth, Ecology, Energy and applied biology	55’852
Humanities	51’309
Physical sciences, Astronomy	48’659
Social sciences	43’164
Computer and information sciences	37’948
Engineering	36’516
Chemistry	28’856
Mathematics	18’437
unknown	105

Table 2. Table 2: Publication counts by typology (genre).

Publication genre	Publication counts
Journal article	470’434
Book chapter	45’795
Proceedings	29’191
Other	18’127
Pre-print	9’636
Book	3’354

Table 3. Table 3: Publication counts by access status.

Access status	Publication counts
Closed	193’222
Publisher-repository	183’554
Repository	102’907
Publisher	96’534

Table 4. Table 4: Publication counts by open access status.

Open access status	Publication counts
Gold	132’435
Green	107’370
Hybrid	92’824
Bronze	51’329

Table 5. Table 5: Descriptive statistics for the dependent variable and a set of publication and author level controls.

Statistic	Cited by Count	Number of Authors	Number of References
Min.	0	0	0
1st Qu.	0	2	1
Median	2	5	23
Mean	11	6.9	32
3rd Qu.	10	8	46
Max.	63’705	100	4’083

Table 6. Table 6: Descriptive statistics for open science indicators.

Variable	FALSE Count	TRUE Count	NA Count
Pre-print match	531’774	44’763	0
Software used	202’336	136’300	237’901
Software created	318’136	20’500	237’901
Software shared	330’305	8’331	237’901
Data used	81’890	256’201	238’446
Data created	234’950	103’141	238’446
Data shared	303’080	35’011	238’446
Is OA	193’222	382’995	320

Table 7. Table 7: Results (base model)

	Dependent variable:
	n_cit_tot_log
	OLS	robust
		linear
	(1)	(2)
n_authors_log	${0.399}^{\ast \ast \ast }$	${0.386}^{\ast \ast \ast }$
	$\left(0.002\right)$	$\left(0.002\right)$
n_references_log	${0.454}^{\ast \ast \ast }$	${0.468}^{\ast \ast \ast }$
	$\left(0.001\right)$	$\left(0.001\right)$
year	$-{0.330}^{\ast \ast \ast }$	$-{0.324}^{\ast \ast \ast }$
	$\left(0.002\right)$	$\left(0.002\right)$
month	$-{0.023}^{\ast \ast \ast }$	$-{0.023}^{\ast \ast \ast }$
	$\left(0.0005\right)$	$\left(0.0004\right)$
C(pre-print_match)	${0.204}^{\ast \ast \ast }$	${0.195}^{\ast \ast \ast }$
	$\left(0.005\right)$	$\left(0.005\right)$
C(software_created)	${0.028}^{\ast \ast \ast }$	${0.029}^{\ast \ast \ast }$
	$\left(0.007\right)$	$\left(0.007\right)$
C(software_shared)	${0.063}^{\ast \ast \ast }$	${0.063}^{\ast \ast \ast }$
	$\left(0.011\right)$	$\left(0.011\right)$
C(data_created)	${0.127}^{\ast \ast \ast }$	${0.122}^{\ast \ast \ast }$
	$\left(0.004\right)$	$\left(0.004\right)$
C(data_shared)	${0.132}^{\ast \ast \ast }$	${0.127}^{\ast \ast \ast }$
	$\left(0.006\right)$	$\left(0.006\right)$
Constant	${666.250}^{\ast \ast \ast }$	${653.919}^{\ast \ast \ast }$
	$\left(4.173\right)$	$\left(4.081\right)$
Observations	337,929	337,929
R2	0.450
Adjusted R2	0.450
Residual Std. Error (df = 337919)	0.970	0.880
F Statistic	30,682.610∗∗∗ (df = 9; 337919)
Note:	∗p0.1; ∗∗p0.05; ∗∗∗p0.01

Table 8. Table 8: Results (full model)

	Dependent variable:
	n_cit_tot_log
	OLS	robust
		linear
	(1)	(2)
n_authors_log	${0.346}^{\ast \ast \ast }$	${0.326}^{\ast \ast \ast }$
	$\left(0.003\right)$	$\left(0.002\right)$
n_references_log	${0.379}^{\ast \ast \ast }$	${0.382}^{\ast \ast \ast }$
	$\left(0.001\right)$	$\left(0.001\right)$
year	$-{0.327}^{\ast \ast \ast }$	$-{0.316}^{\ast \ast \ast }$
	$\left(0.002\right)$	$\left(0.002\right)$
month	$-{0.023}^{\ast \ast \ast }$	$-{0.022}^{\ast \ast \ast }$
	$\left(0.0004\right)$	$\left(0.0004\right)$
C(genre)book	${0.635}^{\ast \ast \ast }$	${0.604}^{\ast \ast \ast }$
	$\left(0.032\right)$	$\left(0.032\right)$
C(genre)book-chapter	$-{0.686}^{\ast \ast \ast }$	$-{0.668}^{\ast \ast \ast }$
	$\left(0.010\right)$	$\left(0.010\right)$
C(genre)other	$-{0.541}^{\ast \ast \ast }$	$-{0.502}^{\ast \ast \ast }$
	$\left(0.013\right)$	$\left(0.013\right)$
C(genre)preprint	$-{1.392}^{\ast \ast \ast }$	$-{1.423}^{\ast \ast \ast }$
	$\left(0.011\right)$	$\left(0.011\right)$
C(genre)proceedings	$-{0.534}^{\ast \ast \ast }$	$-{0.541}^{\ast \ast \ast }$
	$\left(0.010\right)$	$\left(0.010\right)$
C(preprint_match)	${0.174}^{\ast \ast \ast }$	${0.177}^{\ast \ast \ast }$
	$\left(0.006\right)$	$\left(0.005\right)$
C(software_used)	$-{0.057}^{\ast \ast \ast }$	$-{0.030}^{\ast \ast \ast }$
	$\left(0.004\right)$	$\left(0.004\right)$
C(software_created)	${0.061}^{\ast \ast \ast }$	${0.053}^{\ast \ast \ast }$
	$\left(0.007\right)$	$\left(0.007\right)$
C(software_shared)	${0.127}^{\ast \ast \ast }$	${0.116}^{\ast \ast \ast }$
	$\left(0.011\right)$	$\left(0.011\right)$
C(data_used)	${0.148}^{\ast \ast \ast }$	${0.162}^{\ast \ast \ast }$
	$\left(0.005\right)$	$\left(0.005\right)$
C(data_created)	${0.076}^{\ast \ast \ast }$	${0.075}^{\ast \ast \ast }$
	$\left(0.004\right)$	$\left(0.004\right)$
C(data_shared)	${0.134}^{\ast \ast \ast }$	${0.122}^{\ast \ast \ast }$
	$\left(0.006\right)$	$\left(0.006\right)$
C(is_oa)	${0.083}^{\ast \ast \ast }$	${0.075}^{\ast \ast \ast }$
	$\left(0.005\right)$	$\left(0.005\right)$
C(lang_reduce)fr	$-{0.584}^{\ast \ast \ast }$	$-{0.538}^{\ast \ast \ast }$
	$\left(0.007\right)$	$\left(0.006\right)$
C(lang_reduce)other	$-{0.562}^{\ast \ast \ast }$	$-{0.515}^{\ast \ast \ast }$
	$\left(0.014\right)$	$\left(0.014\right)$
C(bso_classification)Biology (fond.)	${0.048}^{\ast \ast \ast }$	${0.063}^{\ast \ast \ast }$
	$\left(0.005\right)$	$\left(0.005\right)$
C(bso_classification)Chemistry	${0.061}^{\ast \ast \ast }$	${0.095}^{\ast \ast \ast }$
	$\left(0.007\right)$	$\left(0.007\right)$
C(bso_classification)Computer and information sciences	${0.107}^{\ast \ast \ast }$	${0.118}^{\ast \ast \ast }$
	$\left(0.008\right)$	$\left(0.008\right)$
C(bso_classification)Earth, Ecology, Energy and applied biology	${0.026}^{\ast \ast \ast }$	${0.051}^{\ast \ast \ast }$
	$\left(0.006\right)$	$\left(0.006\right)$
C(bso_classification)Engineering	${0.066}^{\ast \ast \ast }$	${0.099}^{\ast \ast \ast }$
	$\left(0.007\right)$	$\left(0.007\right)$
C(bso_classification)Humanities	${0.033}^{\ast \ast \ast }$	${0.051}^{\ast \ast \ast }$
	$\left(0.008\right)$	$\left(0.008\right)$
C(bso_classification)Mathematics	$-{0.140}^{\ast \ast \ast }$	$-{0.115}^{\ast \ast \ast }$
	$\left(0.009\right)$	$\left(0.009\right)$
C(bso_classification)Physical sciences, Astronomy	$-{0.102}^{\ast \ast \ast }$	$-{0.059}^{\ast \ast \ast }$
	$\left(0.007\right)$	$\left(0.006\right)$
C(bso_classification)Social sciences	${0.095}^{\ast \ast \ast }$	${0.099}^{\ast \ast \ast }$
	$\left(0.008\right)$	$\left(0.008\right)$
C(bso_classification)unknown	$-0.259$	$-0.221$
	$\left(0.231\right)$	$\left(0.226\right)$
Constant	${660.758}^{\ast \ast \ast }$	${638.442}^{\ast \ast \ast }$
	$\left(3.973\right)$	$\left(3.888\right)$
Observations	337,928	337,928
R2	0.503
Adjusted R2	0.503
Residual Std. Error (df = 337898)	0.922	0.858
F Statistic	11,779.980∗∗∗ (df = 29; 337898)
Note:	∗p0.1; ∗∗p0.05; ∗∗∗p0.01

Table 9. Table 9: Base model, PLOS OSI and FOSM overlap.

	Dependent variable:
	n_cit_tot_log
	OLS	robust
		linear
	(1)	(2)
n_authors_log	${0.433}^{\ast \ast \ast }$	${0.399}^{\ast \ast \ast }$
	$\left(0.027\right)$	$\left(0.027\right)$
n_references_log	${0.333}^{\ast \ast \ast }$	${0.335}^{\ast \ast \ast }$
	$\left(0.031\right)$	$\left(0.031\right)$
year	$-{0.451}^{\ast \ast \ast }$	$-{0.460}^{\ast \ast \ast }$
	$\left(0.020\right)$	$\left(0.019\right)$
month	$-{0.039}^{\ast \ast \ast }$	$-{0.040}^{\ast \ast \ast }$
	$\left(0.005\right)$	$\left(0.004\right)$
C(pre-print_match)	${0.254}^{\ast \ast \ast }$	${0.253}^{\ast \ast \ast }$
	$\left(0.036\right)$	$\left(0.035\right)$
C(software_created)	$-0.069$	$-0.043$
	$\left(0.064\right)$	$\left(0.062\right)$
C(software_shared)	${0.183}^{\ast \ast \ast }$	${0.163}^{\ast \ast }$
	$\left(0.068\right)$	$\left(0.066\right)$
C(data_created)	${0.066}^{\ast \ast }$	${0.055}^{\ast }$
	$\left(0.033\right)$	$\left(0.032\right)$
C(data_shared)	${0.158}^{\ast \ast \ast }$	${0.158}^{\ast \ast \ast }$
	$\left(0.047\right)$	$\left(0.045\right)$
Constant	${912.210}^{\ast \ast \ast }$	${928.924}^{\ast \ast \ast }$
	$\left(39.459\right)$	$\left(38.540\right)$
Observations	2,812	2,812
R2	0.269
Adjusted R2	0.267
Residual Std. Error (df = 2802)	0.832	0.747
F Statistic	114.584∗∗∗ (df = 9; 2802)
Note:	∗p0.1; ∗∗p0.05; ∗∗∗p0.01

Table 10. Table 10: Full model, PLOS OSI and FOSM overlap.

	Dependent variable:
	n_cit_tot_log
	OLS	robust
		linear
	(1)	(2)
n_authors_log	${0.442}^{\ast \ast \ast }$	${0.407}^{\ast \ast \ast }$
	$\left(0.028\right)$	$\left(0.028\right)$
n_references_log	${0.323}^{\ast \ast \ast }$	${0.328}^{\ast \ast \ast }$
	$\left(0.033\right)$	$\left(0.032\right)$
year	$-{0.451}^{\ast \ast \ast }$	$-{0.460}^{\ast \ast \ast }$
	$\left(0.020\right)$	$\left(0.019\right)$
month	$-{0.038}^{\ast \ast \ast }$	$-{0.040}^{\ast \ast \ast }$
	$\left(0.005\right)$	$\left(0.005\right)$
C(pre-print_match)	${0.246}^{\ast \ast \ast }$	${0.242}^{\ast \ast \ast }$
	$\left(0.037\right)$	$\left(0.036\right)$
C(software_used)	$0.022$	$0.016$
	$\left(0.040\right)$	$\left(0.039\right)$
C(software_created)	$-0.084$	$-0.055$
	$\left(0.065\right)$	$\left(0.064\right)$
C(software_shared)	${0.175}^{\ast \ast }$	${0.155}^{\ast \ast }$
	$\left(0.068\right)$	$\left(0.067\right)$
C(data_used)	$-0.002$	$0.008$
	$\left(0.155\right)$	$\left(0.152\right)$
C(data_created)	${0.061}^{\ast }$	$0.053$
	$\left(0.033\right)$	$\left(0.033\right)$
C(data_shared)	${0.152}^{\ast \ast \ast }$	${0.152}^{\ast \ast \ast }$
	$\left(0.047\right)$	$\left(0.046\right)$
C(bso_classification)Biology (fond.)	$0.042$	$0.045$
	$\left(0.037\right)$	$\left(0.037\right)$
C(bso_classification)Chemistry	$-0.048$	$-0.057$
	$\left(0.184\right)$	$\left(0.180\right)$
C(bso_classification)Computer and information sciences	$0.161$	$0.125$
	$\left(0.110\right)$	$\left(0.107\right)$
C(bso_classification)Earth, Ecology, Energy and applied biology	$0.011$	$-0.026$
	$\left(0.068\right)$	$\left(0.067\right)$
C(bso_classification)Engineering	$-0.139$	$-0.096$
	$\left(0.111\right)$	$\left(0.109\right)$
C(bso_classification)Humanities	$0.036$	$0.016$
	$\left(0.080\right)$	$\left(0.078\right)$
C(bso_classification)Mathematics	$0.009$	$0.024$
	$\left(0.166\right)$	$\left(0.163\right)$
C(bso_classification)Physical sciences, Astronomy	$0.153$	$0.199$
	$\left(0.143\right)$	$\left(0.140\right)$
C(bso_classification)Social sciences	${0.201}^{\ast }$	$0.174$
	$\left(0.121\right)$	$\left(0.119\right)$
Constant	${911.482}^{\ast \ast \ast }$	${928.997}^{\ast \ast \ast }$
	$\left(39.570\right)$	$\left(38.807\right)$
Observations	2,812	2,812
R2	0.271
Adjusted R2	0.266
Residual Std. Error (df = 2791)	0.833	0.741
F Statistic	51.954∗∗∗ (df = 20; 2791)
Note:	∗p0.1; ∗∗p0.05; ∗∗∗p0.01

Equations

$$\log (n\_cit\_tot+1)=$$

$$\text{pre-print\_match}+$$

$$\text{software\_created}+\text{software\_shared}+$$

$$\text{data\_created}+\text{data\_shared}$$

$$\log (n\_cit\_tot+1)=$$

$$\text{genre}+\text{pre-print\_match}+$$

$$\text{software\_used}+\text{software\_created}+\text{software\_shared}+$$

$$\text{data\_used}+\text{data\_created}+\text{data\_shared}+$$

$$\text{is\_oa}+\text{bso\_classification}$$

Full text

1 Introduction

Open science is a broad concept and, as defined by the United Nations Educational Scientific and Cultural Organization (UNESCO), combines various movements and practices aiming to make multilingual scientific knowledge openly available, accessible and reusable for everyone  [24]. Within this broad definition of open science is the concept of “open scientific knowledge”, which includes open research outputs and practices associated, or potentially associated, with scientific publications such as open research data, open code and software, and open access to research publications. These open science practices support increased access to, reuse and reproducibility of research, enable new collaborations, and support greater scrutiny of and trust in research. These assumed, and to some extent, evidenced benefits of open science (also known as open research) are among reasons for various stakeholders – researchers, governments, policy makers, research funders and institutions, and journals – to promote open science practices, to various extents. Open science is often promoted through policies and norms of different stakeholder groups, and policies can increase the prevalence of open science practices and outputs – such as increasing the amount of research publications available open access, and more research data being shared openly. However, even more important to policy makers, practitioners, and beneficiaries of open science is the impact, or effects of these open science practices. Evidence of benefits or consequences of open science and associated policies is needed to guide policy implementation and refinement, and justify further investment in open science activities. Monitoring of open science prevalence, and effects, is therefore of increasing importance to stakeholders, including UNESCO member states, which have adopted its recommendation on open science.

The French Open Science Monitor (FOSM; https://frenchopensciencemonitor.esr.gouv.fr) is an initiative of the French Ministry of Higher Education and Research and more specifically the French Committee for Open Science, which ensures implementation of the French National Open Science Policy. To enable the French Committee for Open Science to monitor (measure) the impact of its National Open Science Policy, it introduced the FOSM in 2018, to measure the progress in the adoption of open science practices in France. The first edition of the FOSM in 2018 provided indicators on open access to scientific publications. The 2023 edition introduced indicators on sharing of research data, and sharing of code and software [2]. The FOSM dataset includes scientific publications – journal articles, book chapters, doctoral theses – with at least one French affiliation and a Crossref Digital Object Identifier (DOI), or a Hyper Articles en Ligne (HAL) identifier for publications without a DOI. The inclusion of publications with at least one French affiliation, rather than only publications with a French corresponding affiliation, means the FOSM dataset provides a comprehensive dataset of publications for an entire country, France, approximately 160,000 publications per year. FOSM is based on open data sources and uses open source tools to produce its results. These include (meta)data from Unpaywall, OpenAlex, the Directory of Open Access Journals (DOAJ), PubMed, Crossref, article PDFs and web crawling techniques. The software tools used to produce the FOSM are open source and include GROBID, a tool for extracting structured data (in XML format) from PDFs; DataStet, a tool that identifies mentions of research data and research datasets in publications; and Softcite, which identifies code and software mentions in publications (https://barometredelascienceouverte.esr.gouv.fr/about/methodology). The methodology of FOSM has been shown to converge for detecting countrywide scientific publications, when compared with alternatives using proprietary data sources. [3] The data produced by the FOSM to create its visualizations and indicators are also made available under an open licence (https://barometredelascienceouverte.esr.gouv.fr/about/opendata), and institutions, such as the University of Lorraine, have adapted the Monitor to create local versions.

The FOSM – and initiatives from members of our group such as PLOS Open Science Indicators [12] – is part of a growing number of global initiatives focused on open science monitoring. These initiatives have coalesced around the global Open Science Monitoring Initiative (OSMI; https://open-science-monitoring.org), which launched in 2024 and aims to promote the adoption of common principles [18] and frameworks for open science monitoring globally, along with specifications for their implementation. Monitoring the outcomes (or impacts/consequences) across diverse regions, contexts, and disciplines is equally important to monitoring the prevalence of open science practices. [21] Measuring the outcomes and impacts of open science is also generally more challenging at a large scale, as it involves evaluating qualities and long-term effects of policy and behavioral changes in real-world settings. The PathOS (Open Science Impact Pathways; https://pathos-project.eu) project aimed to improve ways to measure open science impacts, and produced several systematic scoping reviews on the state of the art on the impacts of open science in three areas: academic impacts, societal impacts, and economic impacts. [7, 14, 23]

Particularly relevant to our area of study is the systematic scoping review of academic impacts of open science, Klebel et al. [14] , which identified citations, quality, efficiency, equity, reuse, ethics and reproducibility as areas of academic impact that have been explored – along with identifying gaps in the evidence for academic impacts of open science. Although there are limitations to what citations can tell evaluators and users of research, citations remain an important metric for understanding the impact of research findings, and are associated with higher research quality, and researcher recognition. Further, due to their widespread use, citations can be measured across large corpora of articles. Klebel et al. [14] identified 22 studies reporting an impact of open data (data sharing) on citations; four for pre-prints; and three for open code (code sharing). However, none confirms a causal relationship between open science practices and citation advantages. Of the previous studies on citation advantages for open data, those which explore the effects of sharing research data in a data repository linked to journal articles are most relevant to our research question and methodology. Piwowar and Vision [19] reported a 9% citation advantage in gene expression studies, Henneken and Accomazzi found a 20% increase in astronomy studies [11], and across 531,889 articles published in BMC and PLOS journals, Colavizza et al. [6] found a 25% increase in citations for articles that link to data shared in an external data repository. Fu and Huey reported a 36% citation advantage for articles that share an associated pre-print [10] and Fraser et al., found that after 36 months, articles with an associated pre-print received 1.74 times more citations on average [8]. Citation effects of code sharing are less well studies across large and/or cross-disciplinary corpora but positive associations of code sharing on citations have been found by others. [25, 15] Our previous study published in 2024 complemented and contrasted these results by finding a 20.2% citation advantage for the sharing of pre-print and 4.3% advantage for the sharing of data in a repository. [5] However, we did not find a statistically significant citation advantage for code sharing. In this previous analysis we used the PLOS Open Science Indicators dataset, which is produced in collaboration with DataSeer, and has monitored five open science practices including data, code, and pre-print sharing, as well as, more recently, protocol sharing, and study registration (preregistration). As an ongoing open science monitoring initiative, the PLOS Open Science Indicators dataset is updated periodically as new PLOS and comparator content is published. Our previous study was based on version 5 of the dataset, which included 121,999 articles, the majority of which are PLOS articles. [20]

In this present study, we wanted to explore if similar or different citation effects for the same open science practices we evaluated in our previous study (data sharing, code sharing, pre-print sharing) exist in a larger and more diverse sample of publications. To enable us proceed efficiently we sought openly available data on open science monitoring activities of research publications that could be reused in our citation prediction model. Given the alignment of the indicators measured in the PLOS Open Science Indicators and the indicators included in the FOSM, and its production of reusable open data, we chose the FOSM’s larger and more diverse dataset (https://data.enseignementsup-recherche.gouv.fr/explore/dataset/open-access-monitor-france/information) to explore this question. Furthermore, given the purpose of the FOSM is to inform understanding of open science policy effects and the established national open science policy in France, we assumed the results would be valuable for policy makers, funding agencies, researchers, publishers, institutions, and other stakeholders.

2 Data and Methods

The 2023 edition of the FOSM dataset contained approximately 900’000 publications at the time of our analysis. The indicators of interest include whether data and software were created and shared, and whether there are pre-prints associated with published articles. We seek to answer the research question based on the FOSM dataset, namely, whether and to what extent a citation impact premium is received on average by articles following some or all of the open science practices under consideration. This work follows the methodology, and expands upon the results of a previously published work [6, 5], including PLOS’ Open Science Indicators [20].

To collect the data for analysis, we proceed as follows:

Acquire the FOSM dataset (https://data.enseignementsup-recherche.gouv.fr/explore/dataset/open-access-monitor-france/information). We use the December 30, 2022 snapshot (end of coverage period). This snapshot includes 897’426 entries. Of these, 545’158 are classified as journal articles (60.75%). The publications were published between 2020 and 2022, both years included. The number of publications per year is roughly uniform. 2. 2.

Query OpenAlex (https://openalex.org). for every DOI present in FOSM, and download the full OpenAlex record locally. OpenAlex was queried between July 12 and 15, 2024. The number of entries in FOSM with an OpenAlex match, after the removal of duplicates, is 576’537 (64.24%). This dataset contains 479’700 journal articles (83.2% of the matched total), and is used in what follows. 3. 3.

Add pre-print information from Crossref, using an experimental pre-print-publication relationship dataset created by Crossref. [22] This dataset covers Crossref publications up to the end of August 2023, therefore fully matching the FOSM coverage. The number of entries in this Crossref dataset is 641’950. Of these, 22’303 match with both FOSM and OpenAlex (3.9% of FOSM with an OpenAlex match). The total number of pre-print matches, including also arXiv matches indexed in OpenAlex, is 44’763 (7.8%). This proportion follows known trends in the literature [17].

From OpenAlex, for every entry, we take the month of publication, the number of authors and references, whether there is a pre-print in arXiv, and the citation count at the time of the query (July 2024). From Crossref, we recover possible further matches with other pre-print servers. From FOSM, we take every other variable we use in what follows.

We provide several statistics on the available variables. Please note that the totals may vary since variables differ in the amount of not available values. See Table 6 for NA value counts. Table 1 shows the publication counts for different BSO categories. BSO (Baromètre de la Science Ouverte) is a system to classify scientific domains into macro areas. In FOSM, each publication is assigned a BSO class automatically, see [13] for methodological details. Note that Biology (fond.) stands for fundamental (fondamentale in French), as distinct from applied biology belonging to another class. Next, we show the publication counts by publication typology (genre) (Table 2), showing a large majority for journal articles, by access status (Table 3) and open access status (Table 4, please note the values here refer only to open access publications). Lastly, we provide descriptive statistics for the main dependent variable (citation counts), controls (Table 5), and open science indicator controls (Table 6). Minimum values at zero in Table 5 for the number of authors and references of publications likely hint at missing data limitations from OpenAlex. We dealt with such cases in the models that follow by adding one, in order to avoid dropping observations on control, low-importance variables. We note that the maximum correlation among controls is 0.25 for software and data sharing, thus ruling out issues due to multicollinearity. This is part of standard checks done before model fitting, that include the need to avoid using highly correlated controls which would make the model fitting less reliable.

From now on, we consider as open science indicators (OSI) the following variables of interest: pre-print publication, software shared, data shared. We also include software used and created, and data used and created. Although strictly speaking these are not open science practices, they are useful comparisons for sharing activities.

3 Results

We start by visualizing the temporal trends in OSI adoption, reminding the reader that the FOSM dataset used in our analysis only covers publication from 2020, 2021, and 2022. In Figure 1, we notice how most OSI remain stable and relatively low over time, despite significant levels of data and software use. In Figure 2, we unpack trends by BSO class, noticing some common trends such as higher software creation in Computer Science, and pre-print adoption in Physics and Mathematics.

3.1 Modeling

The base model we use is described in Equation 1, and the full model is described in Equation 2. Variable transformations are shown, numerical variables are given in Italics, and categorical variables are in regular text. Variables are grouped along lines. This model follows closely from our previous work, where an illustration of the assumed causal dependency graph among variable groups is also provided [5]. We remark that, while the variables year and month could also be considered as categorical or fixed effects, the results do not differ substantially. The code to replicate these and all other models is provided. We also notice that, with respect to our previous work, we have no ability to control for the mean h-index of the authors at the time of publication, and we use the BSO classification instead of the Fields of Research classification codes (https://www.arc.gov.au/manage-your-grant/classification-codes-rfcd-seo-and-anzsic-codes). At the same time, we add further variables such as software/data used or created, publication genre (e.g. Journal article, Book chapter), and open access status. The absence of the h-index control is the most significant limitation of our work here, in terms of its comparability to our previous work, as it constitutes a proxy for author success or reputation at the time of publication, which might also correlate with resource availability. To confirm the absence of discrepancies, we provide results on a small set of journal articles that are part of both PLOS OSI and FOSM in the Appendix.

The codebase contains a variety of post-estimation diagnostics and alternative models, including residual behavior, influential observations, multicollinearity, model specification checks. A Breusch–Pagan test strongly rejects homoskedasticity (BP = $9888.8$, p < $0.001$). We therefore also add a robust OLS model throughout. The codebase also contains ways to model with robust and clustered standard errors as further checks. These are omitted here as they align with the provided results.

[TABLE]

[TABLE]

Starting with the base model in Table LABEL:tab:model_base, we provide results for an Ordinary Least Squares (OLS) model and a robust linear model as a comparison. Boolean variables use “False” as reference category. The results are aligned and show a relatively high explained variance with the base model having $R^{2}=.45$. The model shows expected trends with respect to common controls (authors, references, year of publication), and interesting novel trends with respect to OSI. The OSI show that there is a significant and positive correlation of pre-prints ($22.6\%$), sharing data ($14.1\%$) and sharing software ($6.5\%$) with increased citations. These percentage changes for log-linear relationships are calculated as follows: $(\exp(.204)-1)\times 100\approx 22.6\%$. Our next question is whether these results hold when we account for large disciplinary variations in the adoption of open science practices, as well as further controls which we add next.

In the full model we propose, in Table LABEL:tab:model_full, we add the following variables: publication genre, software/data used, open access status, language, and BSO classification. We take the categories with most publications as reference category, specifically: ‘journal-article’ for genre, ‘English’ for language, and ‘Medical research’ for BSO classification. Our model confirms previous trends, namely pre-print match contributing to a significant and positive correlation of $19\%$ with citation counts, software sharing of $13.5\%$, and data sharing of $14.3\%$. We also find positive effects from software/data creation. Publications where data is used tend to be more cited, whilst those when software is used slightly less so. Publications in English are significantly more cited than publications in other languages. Disciplinary differences (with respect to the baseline Medical research), and publication genre differences (with respect to the baseline journal article), also follow known trends.

While open access (OA) status is not a relevant indicator for the PLOS OSI dataset given all content is OA, OA status is an important aspect of the FOSM, with OA adoption reaching 65.6% in 2022 in French publications (https://frenchopensciencemonitor.esr.gouv.fr/). We included OA status in our model and can observe a $8.6\%$ citation increase correlated with publications that are OA compared to those that are closed access. For a review on the citation impact of open access status, see [16]. OA status can be explored in more depth considering OA typologies using our data and code. [4]

These results are robust to a variety of modeling changes, which we include in our accompanying repository.

Lastly, we provide a summary table with the percentage change on citation counts linked to each OSI available, organizing publications by scientific domain (BSO class) and using only the main publication genre: journal articles. Percentage changes are calculated exactly as we did above, from discipline-specific log-linear models, with the reference category being publications within the same BSO class not exhibiting the given open science indicator. These results should be more easily readable than those provided above, when interested in differences across scientific domains. We fit a full model specification for every BSO class, and provide results in Figure 3. While these results confirm the trends discussed above, they also highlight significant variations across scientific domains. For example, the open access status is positively correlated to citation counts only in Medicine and Biology, and negatively elsewhere.

4 Discussion and Conclusions

In this work, we have explored the FOSM dataset and investigated whether three open science indicators (OSI) – data sharing, code sharing, pre-print posting – are correlated with a citation advantage received by the publications that exhibit these open science practices. To our knowledge, this is the largest scale analysis of these open science practices and citations to date, and the first such study offering a nationwide (French) perspective on this topic. We use over half a million publications from all scientific domains, and find a positive correlation of these open science practices with citation counts. This study is observational and limited by the scope of the analysis and the available data, therefore its results should be taken with caution. As a reanalysis of existing data, which have been generated with machine learning approaches, our results are also limited by the accuracy of the original data, in particular its detection of shared data and code for articles. The accuracy (F1 scores) of the machine learning methodology of the FOSM for detection of data and code use, creation, and sharing range from 82.19 to 99.03 are fully reported in [2]. We are in no position to establish causality nor to control for additional, confounding effects. Significant confounding effects that we could not consider are the quality of the research, and the reputation of the authors (as measured by h-index) and of the venue of publication. A more developed discussion of these and other limitations of this approach to measuring citation effects is provided in [5]. In theory, the observed correlation between increased citations and these open science practices in our analysis are cumulative, which would suggest a substantial average increase – $46.8\%$ – in citations where an article exhibited all three open science practices. However, such an assumption should be made with caution given the aforementioned limitations.

Our finding of a significant positive effect of pre-print posting on citations, of $19\%$ in this study, confirms results of previous studies. [9, 5] Our finding of a significant positive effect of software sharing of $13.5\%$ contrasts the result of our previous study, which found no statistically significant increase in citations correlated with code (software) sharing. [5] This may be due to the larger and more diverse, with respect to research topics and disciplines, dataset used in this study compared to our previous work. However, other studies [25] have found as much as a three-fold increase in citations associated with code sharing. Our finding of a significant positive effect of data sharing of $14.3\%$ confirms a significant positive effect observed in two previous studies, which found increases in citations correlated with data sharing in a repository of $25.3\%$ when analysing publications in PLOS and BMC (BioMed Central) and $4.3\%$ when analyzing publications in PLOS OSI. [6, 5] The larger effect of OSI on citations observed in the current study may be due to the larger and more diverse corpus of publications, which includes, for example, a higher proportion of content from the Humanities and Physical Sciences than these previous two studies. Also, the FOSM publication corpus includes content that is not open access, in contrast to these previous two studies. The source of publications and citation counts (OpenAlex) in the current study is different to our previous studies, which relied on PubMed Central. OpenAlex is emerging as a credible alternative to established databases for bibliometric analyses but may lag behind in the number of indexed references (and as a consequence, citations) – at least when compared to the commercial database, Scopus. [1]

While we have focused our interpretation on average citation effects across the analysed dataset, we can observe some notable differences in these averages when results are segmented by BSO class (research topic or discipline), in Figure 3. For example, data sharing is correlated with a $14.3\%$ increase in citations on average in our analysis but with publications classified as Medical research, we observe a larger, $34.9\%$ increase. Also, the effect of pre-printing is particularly high in the Humanities, where pre-printing is correlated with a $62.5\%$ increase, compared to $19\%$ in the dataset overall. We can only speculate here on the reasons for these differences. Public data sharing in Medical research can be more challenging compared to fields that do not produce or analyze sensitive data, and so sharing these data comes with additional processing, security, and management costs. Where shared datasets are available, such as from important clinical trials, then it is possible they will have greater reuse value, in turn driving more citations. In other words, we could speculate that the additional effort on behalf of researchers to share high value datasets is rewarded with greater impact of their work. In the Humanities, our sample of publications is smaller than other disciplines covered in the dataset, but we can speculate that cultural and temporal differences in publishing and citation in the Humanities relative to other disciplines are a factor. That is, publication output and speed tends to be slower than the life and medical sciences, and total numbers of citations lower. Therefore, the small proportion of publications that do have a pre-print available could potentially benefit from greater visibility and therefore citations, amplifying the observed average effect of pre-printing relative to disciplinary norms. Conversely, we observe a small negative correlation of data sharing (-$1.4\%$) in the Humanities.

Regarding the larger difference we observed in the citation effect of data sharing compared to our previous study finding a $4.3\%$ positive correlation, differences in how open science practices are defined and measured between open science monitoring methods could be a factor. In [5], the positive correlation was observed for a specific method of data sharing, data sharing in a repository linked to the article (as opposed to sharing data in online supplementary material). There are some subtle but important differences in how open science practices are defined and, consequently, measured, in different open science monitoring initiatives and software tools. Systematically mapping OSI definitions and measures across multiple studies is outside of the scope of the current work, although we provide a small-scale mapping of a subset of publications between our current and previous study in the Appendix. This issue highlights the need for development of common frameworks for, and consensus on, methods for open science monitoring, using open methods and data.

With the above limitations, this study adds to the evidence that OSI (open science practices) are associated with increased academic impact of research through citations, and provide for the first time, to our knowledge, a perspective on this question for an entire country, France. Open science monitoring is still in its relative infancy, but France provides a valuable case study having implemented a comprehensive open science policy for research publications and outputs since 2018 and latterly introducing a means to monitor its progress. The FOSM has helped to demonstrate that the prevalence of open science practices such as open access to publications, open data, and open code/software have increased since the implementation of their national policy. Open access to publications has, for example, increased from $38\%$ in 2018 to $64\%$ in 2023 and open data from and $16\%$ to $25\%$ in the same period of time (https://frenchopensciencemonitor.esr.gouv.fr). Our findings complement these observations by providing policy makers and other stakeholders with evidence that, as well as increasing prevalence of open science, there open science policies may be a benefit in terms of greater impact of research through open science practices as measured by citations. Future research could seek to understand if open science practices lead to other (non-citation based) academic impacts of research, such as whether they lead to increased (re)use of research outputs, improved reproducibility, economic impact, or greater trust in research outputs. More work on determining causal mechanisms of changes in academic impacts of open science is also needed.

Data and Code Availability

Please find all data and code to reproduce our results at https://doi.org/10.6084/m9.figshare.27822663. [4]

Acknowledgments

For their support in using the French Open Science Monitor dataset, we thank Eric Jeangirard and Anne L’Hote at the French Ministry of Higher Education and Research, Laetitia Bracco at the University of Lorraine and Laurent Romary at the National Institute for Research in Digital Science and Technology (Inria). We also thank Marin Dacos and Arianna Caporali at the French Ministry of Higher Education and Research for participating in discussions with PLOS that contributed to the conceptualization of this research.

Author contributions

GC: Conceptualization, Data curation, Formal analysis, Investigation, Methodology, Software, Supervision, Writing – original draft, Writing – review & editing.

LC: Formal Analysis, Visualization, Writing – review & editing.

IH: Conceptualization, Funding acquisition, Methodology, Project administration, Resources, Supervision, Writing – original draft, Writing – review & editing.

Competing interests

Two of the authors (LC and IH) were at the time of publication employed by PLOS, which produces PLOS Open Science Indicators. PLOS (IH) is a founding member and initiator of the Open Science Monitoring Initiative (OSMI). GC declares no competing interests.

Funding information

PLOS provided funding for data acquisition, modeling, and analysis, and had a role in the design, analysis, and preparation of the study. PLOS also provided support in the form of salaries for authors LC and IH.

Appendix

We report here on an analysis using the same models but focused on the overlap between FOSM and the PLOS OSI dataset (version 7). The overlap consists of 3120 publications, from PLOS and the control group. Note that some control variables (genre, open access status) had to be omitted due to missing factors. The results align with the findings of [5] and with the findings of this analysis, showing larger effects for data and software sharing.

Please note there are some differences in definitions and measures in the FOSM compared to PLOS OSI, reflecting the different methods used by the FOSM, and PLOS/DataSeer teams. For reference, the FOSM uses the following definitions for data/software use, creation, and sharing [2]:

• •

Data/software use refers to publications that mention the usage of at least one dataset/software.

• •

Data/software created refers to publications that mention the usage and the production of at least one of their datasets/software.

• •

Data/software shared refers to publications that mention the usage, the production and the sharing of at least one of their datasets/software.

While our measures of code sharing are well aligned across FOSM and PLOS OSI, in our previous study of PLOS OSI datasets, for data sharing, we assessed the effects of sharing research data on citations in a more specific way, that is, sharing via an online data repository [5].