Crowdsourcing the State of the Art(ifacts)

TL;DR

This paper introduces a novel crowdsourced reuse graph method to identify the state-of-the-art in research fields, demonstrated through a study of software engineering papers showing widespread reuse.

Contribution

It presents a new, less effortful approach to determine the research frontier using crowdsourced reuse graphs, improving over traditional citation-based methods.

Findings

Over 1,600 reuse instances found in 170 SE papers

Reuse is more prevalent than previously believed

The method is easier to build and verify than existing approaches

Abstract

In any field, finding the "leading edge" of research is an on-going challenge. Researchers cannot appease reviewers and educators cannot teach to the leading edge of their field if no one agrees on what is the state-of-the-art. Using a novel crowdsourced "reuse graph" approach, we propose here a new method to learn this state-of-the-art. Our reuse graphs are less effort to build and verify than other community monitoring methods (e.g. artifact tracks or citation-based searches). Based on a study of 170 papers from software engineering (SE) conferences in 2020, we have found over 1,600 instances of reuse; i.e., reuse is rampant in SE research. Prior pessimism about a lack of reuse in SE research may have been a result of using the wrong methods to measure the wrong things.