From Academia to Software Development: Publication Citations in Source Code Comments

TL;DR

This study analyzes how academic publications influence software development by detecting citations in source code comments across large open source repositories, revealing insights into knowledge transfer from academia to industry.

Contribution

It introduces an automated NER-based method to detect publication citations in code comments and provides a large-scale analysis of their impact on open source software.

Findings

Academic citations are mainly from journals.

Algorithm knowledge is most transferred from publications.

Most references are from science-related repositories.

Abstract

Academic publications have been evaluated in terms of their impact on research communities based on many metrics, such as the number of citations. On the other hand, the impact of academic publications on industry has been rarely studied. This paper investigates how academic publications contribute to software development by analyzing publication citations in source code comments in open source software repositories. We propose an automated approach for detecting academic publications based on Named Entity Recognition, and achieve 0.90 in $F_1$ as detection accuracy. We conduct a large-scale study of publication citations with 319,438,977 comments collected from 25,925 active repositories written in seven programming languages. Our findings indicate that academic publications can be knowledge sources for software development. These referenced publications are particularly from journals. In terms of knowledge transfer, algorithm is the most prevalent type of knowledge transferred from the publications, with proposed formulas or equations typically implemented in methods or functions in source code files. In a closer look at GitHub repositories referencing academic publications, we find that science-related repositories are the most frequent among GitHub repositories with publication citations, and that the vast majority of these publications are referenced by repository owners who are different from the publication authors. We also find that referencing older publications can lead to potential issues related to obsolete knowledge.