What is the Impact of Releasing Code with Publications? Statistics from the Machine Learning, Robotics, and Control Communities

TL;DR

Releasing code alongside research papers significantly boosts scientific impact and reproducibility, with increasing adoption across machine learning, robotics, and control communities over recent years.

Contribution

This study provides statistical evidence on the positive impact of code sharing in scientific research across three major communities, highlighting trends and correlations with research impact.

Findings

Code sharing has doubled in major conferences from 2016 to 2021.

High-impact papers are more likely to include open-source code.

Popular code repositories correlate with higher citation counts.

Abstract

Open-sourcing research publications is a key enabler for the reproducibility of studies and the collective scientific progress of a research community. As all fields of science develop more advanced algorithms, we become more dependent on complex computational toolboxes -- sharing research ideas solely through equations and proofs is no longer sufficient to communicate scientific developments. Over the past years, several efforts have highlighted the importance and challenges of transparent and reproducible research; code sharing is one of the key necessities in such efforts. In this article, we study the impact of code release on scientific research and present statistics from three research communities: machine learning, robotics, and control. We found that, over a six-year period (2016-2021), the percentages of papers with code at major machine learning, robotics, and control conferences have at least doubled. Moreover, high-impact papers were generally supported by open-source codes. As an example, the top 1% of most cited papers at the Conference on Neural Information Processing Systems (NeurIPS) consistently included open-source codes. In addition, our analysis shows that popular code repositories generally come with high paper citations, which further highlights the coupling between code sharing and the impact of scientific research. While the trends are encouraging, we would like to continue to promote and increase our efforts toward transparent, reproducible research that accelerates innovation -- releasing code with our papers is a clear first step.