Does Functional Package Management Enable Reproducible Builds at Scale? Yes

TL;DR

This large-scale study demonstrates that functional package management with Nix achieves high reproducibility and rebuildability rates across hundreds of thousands of packages, supporting scalable reproducible builds.

Contribution

First extensive analysis of reproducibility at scale in Nix, revealing high success rates and identifying key causes of unreproducibility in large package repositories.

Findings

Reproducibility rates between 69% and 91% with upward trend.

Rebuildability exceeds 99%.

Approximately 15% of failures due to embedded build dates.

Abstract

Reproducible Builds (R-B) guarantee that rebuilding a software package from source leads to bitwise identical artifacts. R-B is a promising approach to increase the integrity of the software supply chain, when installing open source software built by third parties. Unfortunately, despite success stories like high build reproducibility levels in Debian packages, uncertainty remains among field experts on the scalability of R-B to very large package repositories. In this work, we perform the first large-scale study of bitwise reproducibility, in the context of the Nix functional package manager, rebuilding 709 816 packages from historical snapshots of the nixpkgs repository, the largest cross-ecosystem open source software distribution, sampled in the period 2017-2023. We obtain very high bitwise reproducibility rates, between 69 and 91% with an upward trend, and even higher rebuildability rates, over 99%. We investigate unreproducibility causes, showing that about 15% of failures are due to embedded build dates. We release a novel dataset with all build statuses, logs, as well as full ''diffoscopes'': recursive diffs of where unreproducible build artifacts differ.