The Promise and Reality of Continuous Integration Caching: An Empirical Study of Travis CI Builds

TL;DR

This empirical study investigates how CI caching is adopted and maintained in practice on Travis CI, revealing low adoption rates, common challenges, and the impact on build times across a large sample of projects.

Contribution

It provides the first large-scale analysis of CI caching adoption, challenges, and maintenance activities in real-world Travis CI projects, highlighting gaps between expectations and practice.

Findings

Only 30% of projects adopt CI caching.

Nearly half of non-adopters accepted caching pull requests.

One-third of projects experienced significant build-time reductions.

Abstract

Continuous Integration (CI) provides early feedback by automatically building software, but long build durations can hinder developer productivity. CI services use caching to speed up builds by reusing infrequently changing artifacts, yet little is known about how caching is adopted in practice and what challenges it entails. In this paper, we conduct a large-scale empirical study of CI caching in Travis CI, analyzing 513,384 builds from 1,279 GitHub projects. We find that only 30% of projects adopt CI caching, and early adopters are typically more mature, with more dependencies, commits, and longer CI lifespans. To understand non-adoption, we submit pull requests enabling caching in non-adopting projects, and nearly half are accepted or merged. Developer feedback indicates that non- or late adoption mainly results from limited awareness of CI caching support. We further study cache maintenance and identify five common activities, performed by 24% of cache-enabled projects. While one-third of projects see substantial build-time reductions, cache uploads occur in 97% of builds, and 27% of projects contain stale cached artifacts. An analysis of reported caching issues shows developers mainly struggle with corrupted or outdated caches and request broader caching features. Overall, CI caching does not benefit all projects, requires ongoing maintenance, and is more complex in practice than many developers expect.