From First Use to Final Commit: Studying the Evolution of Multi-CI Service Adoption

TL;DR

This study analyzes how open-source projects adopt, switch between, and maintain multiple CI services over time, revealing that nearly 20% of projects use multiple services, often migrating between them.

Contribution

It provides the first large-scale empirical analysis of multi-CI service adoption and migration patterns in open-source projects, highlighting the complexity of CI ecosystem evolution.

Findings

Nearly 20% of projects co-adopt or migrate between CI services.

Migration patterns indicate frequent switching and multi-service usage.

Maintenance activity varies significantly across different CI services.

Abstract

Continuous Integration (CI) services, such as GitHub Actions and Travis CI, are widely adopted in open-source development to automate testing and deployment. Though existing research often examines individual services in isolation, it remains unclear how projects adopt and transition between multiple services over time. To understand how CI adoption is evolving across services, we present a preliminary study analyzing the historical CI adoption of 18,924 Java projects hosted on GitHub between January 2008 and December 2024, adopting at least one of eight CI services, namely Travis CI, AppVeyor, CircleCI, Azure Pipelines, GitHub Actions, Bitbucket, GitLab CI, and Cirrus CI. Specifically, we investigate: (1) how frequently CI services are co-adopted or replaced, and (2) how maintenance activity varies across different services. Our analysis shows that the use of multiple CI services within the same project is a recurring pattern observed in nearly one in five projects, often reflecting migration across CI services. Our study is among the first to examine multi-CI adoption in practice, offering new insights for future research and highlighting the need for strategies and tools to support service selection, coordination, and migration in evolving CI environments.