An Empirical Study of Complexity, Heterogeneity, and Compliance of GitHub Actions Workflows

TL;DR

This study empirically analyzes GitHub Actions workflows in open-source projects to understand their complexity, heterogeneity, and compliance with best practices, revealing areas for improvement and guiding better CI pipeline design.

Contribution

It provides the first large-scale empirical analysis of GHA workflows, identifying common patterns, complexities, and compliance issues across multiple programming languages.

Findings

Identification of prevalent workflow complexity patterns

Analysis of heterogeneity in workflow structures

Assessment of adherence to GHA best practices

Abstract

Continuous Integration (CI) has evolved from a tooling strategy to a fundamental mindset in modern CI engineering. It enables teams to develop, test, and deliver software rapidly and collaboratively. Among CI services, GitHub Actions (GHA) has emerged as a dominant service due to its deep integration with GitHub and a vast ecosystem of reusable workflow actions. Although GHA provides official documentation and community-supported best practices, there appears to be limited empirical understanding of how open-source real-world CI workflows align with such practices. Many workflows might be unnecessarily complex and not aligned with the simplicity goals of CI practices. This study will investigate the structure, complexity, heterogeneity, and compliance of GHA workflows in open-source software repositories. Using a large dataset of GHA workflows from Java, Python, and C++ repositories, our goal is to (a) identify workflow complexities, (b) analyze recurring and heterogeneous structuring patterns, (c) assess compliance with GHA best practices, and (d) uncover differences in CI pipeline design across programming languages. Our findings are expected to reveal both areas of strong adherence to best practices and areas for improvement where needed. These insights will also have implications for CI services, as they will highlight the need for clearer guidelines and comprehensive examples in CI documentation.