Local Software Buildability across Java Versions (Registered Report)

TL;DR

This study empirically investigates how the increasing Java versions affect the success rate of building open-source Java projects, revealing trends and factors influencing build failures across different environments.

Contribution

It provides the first large-scale empirical analysis of Java build success rates across multiple versions, identifying key factors and failure reasons.

Findings

Build failure rates increase with newer Java versions

Certain build tools and wrappers improve success rates

Common failure reasons include compatibility and configuration issues

Abstract

Context: Downloading the source code of open-source Java projects and building them on a local computer using Maven, Gradle, or Ant is a common activity performed by researchers and practitioners. Multiple studies so far found that about 40-60% of such attempts fail. Our experience from the last years suggests that the proportion of failed builds rises continually even further. Objective: First, we would like to empirically confirm our hypothesis that with increasing Java versions, the percentage of build-failing projects tends to grow. Next, nine supplementary research questions are proposed, related mainly to the proportions of failing projects, universal version compatibility, failures under specific JDK versions, success rates of build tools, wrappers, and failure reasons. Method: We will sample 2,500 random pure-Java projects having a build configuration file and fulfilling basic quality criteria from GitHub. We will try to automatically build every project in containers with Java versions 6 to 23 installed. Success or failure will be determined by exit codes, and standard output and error streams will be saved. A majority of the analysis will be performed automatically using reproducible scripts.