Reliabuild: Searching for High-Fidelity Builds Using Active Learning

TL;DR

Reliabuild is an auto-tuning framework that efficiently explores complex software build configurations using active learning, significantly improving the identification of successful high-fidelity builds over random sampling.

Contribution

It introduces Reliabuild, a novel auto-tuning framework with models and adaptive sampling to efficiently find high-quality build configurations in large, complex software dependency spaces.

Findings

Selects 3X more good configurations than random sampling

Finds all high-fidelity builds with half the samples of random sampling

Learns package compatibility statistics to aid solvers

Abstract

Modern software is incredibly complex. A typical application may comprise hundreds or thousands of reusable components. Automated package managers can help to maintain a consistent set of dependency versions, but ultimately the solvers in these systems rely on constraints generated by humans. At scale, small errors add up, and it becomes increasingly difficult to find high-fidelity configurations. We cannot test all configurations, because the space is combinatorial, so exhaustive exploration is infeasible. In this paper, we present Reliabuild, an auto-tuning framework that efficiently explores the build configuration space and learns which package versions are likely to result in a successful configuration. We implement two models in Reliabuild to rank the different configurations and use adaptive sampling to select good configurations with fewer samples. We demonstrate Reliabuild's effectiveness by evaluating 31,186 build configurations of 61 packages from the Extreme-scale Scientific Software Stack(E4S). Reliabuild selects good configurations efficiently. For example, Reliabuild selects 3X the number of good configurations in comparison to random sampling for several packages including Abyss, Bolt, libnrm, OpenMPI. Our framework is also able to select all the high-fidelity builds in half the number of samples required by random sampling for packages such as Chai, OpenMPI, py-petsc4py, and slepc. We further use the model to learn statistics about the compatibility of different packages, which will enable package solvers to better select high-fidelity build configurations automatically.