Towards Reproducible Research: Automatic Classification of Empirical Requirements Engineering Papers

TL;DR

This paper presents ERRC, an NLP and machine learning-based classifier that automatically identifies requirements engineering and empirical research papers to promote reproducibility in software engineering.

Contribution

It introduces a novel supervised classification method for automatically detecting empirical RE papers, improving over keyword-based approaches.

Findings

ERRC outperforms baseline methods in most cases

High accuracy in classifying empirical RE papers

Facilitates easier identification of reproducible research

Abstract

Research must be reproducible in order to make an impact on science and to contribute to the body of knowledge in our field. Yet studies have shown that 70% of research from academic labs cannot be reproduced. In software engineering, and more specifically requirements engineering (RE), reproducible research is rare, with datasets not always available or methods not fully described. This lack of reproducible research hinders progress, with researchers having to replicate an experiment from scratch. A researcher starting out in RE has to sift through conference papers, finding ones that are empirical, then must look through the data available from the empirical paper (if any) to make a preliminary determination if the paper can be reproduced. This paper addresses two parts of that problem, identifying RE papers and identifying empirical papers within the RE papers. Recent RE and empirical conference papers were used to learn features and to build an automatic classifier to identify RE and empirical papers. We introduce the Empirical Requirements Research Classifier (ERRC) method, which uses natural language processing and machine learning to perform supervised classification of conference papers. We compare our method to a baseline keyword-based approach. To evaluate our approach, we examine sets of papers from the IEEE Requirements Engineering conference and the IEEE International Symposium on Software Testing and Analysis. We found that the ERRC method performed better than the baseline method in all but a few cases.