Predicting Health Indicators for Open Source Projects (using Hyperparameter Optimization)

TL;DR

This study demonstrates that hyperparameter optimization significantly improves the accuracy of predicting health indicators in open-source projects using large-scale GitHub data.

Contribution

It is the first large-scale study to apply hyperparameter optimization to predict multiple project health indicators accurately.

Findings

Hyperparameter tuning reduces prediction errors substantially.

Traditional algorithms like KNN and SVR have high error rates without optimization.

Large-scale data enables effective prediction of project health status.

Abstract

Software developed on public platform is a source of data that can be used to make predictions about those projects. While the individual developing activity may be random and hard to predict, the developing behavior on project level can be predicted with good accuracy when large groups of developers work together on software projects. To demonstrate this, we use 64,181 months of data from 1,159 GitHub projects to make various predictions about the recent status of those projects (as of April 2020). We find that traditional estimation algorithms make many mistakes. Algorithms like $k$-nearest neighbors (KNN), support vector regression (SVR), random forest (RFT), linear regression (LNR), and regression trees (CART) have high error rates. But that error rate can be greatly reduced using hyperparameter optimization. To the best of our knowledge, this is the largest study yet conducted, using recent data for predicting multiple health indicators of open-source projects.