Reliability of AI Bots Footprints in GitHub Actions CI/CD Workflows

TL;DR

This study analyzes the reliability of AI bots in GitHub CI/CD workflows, revealing agent-dependent success rates and the impact of AI contribution frequency on workflow success.

Contribution

It provides the first large-scale empirical analysis of AI bot reliability in CI/CD workflows, including a taxonomy of failure categories and trend observations.

Findings

Copilot and Codex have success rates of ~93% and ~94%.

Higher AI contribution frequency correlates with lower workflow success.

Identified 13 failure categories and observed shifts over time.

Abstract

Continuous Integration and Deployment (CI/CD) workflows are central to modern software delivery, yet the reliability of agentic AI bots operating within these workflows remain underexplored. Using pull requests (PRs), commits, and repositories from the AIDev dataset, we retrieved associated CI/CD workflow runs via the GitHub Actions API and analyzed 61,837 runs from 2,355 repositories, all triggered by PRs generated by five AI bots: Claude, Devin, Cursor, Copilot, and Codex. We observed substantial agent-dependent differences in workflow reliability, with Copilot and Codex achieving the highest success rates ~93% and ~94% respectively. At the repository level, we find a negative correlation between AI agent contribution frequency and workflow success rate, suggesting that a higher frequency of Agentic PRs may hinder CI/CD workflow reliability. We defined a taxonomy of 13 categories against 3,067 agentic PRs whose associated workflows failed, and observed a trend analysis that indicates visually observable shifts from functional to non-functional PR categories over time, although these trends are not statistically significant. Our findings motivate the need for actionable guidance on integrating AI agents into CI/CD workflows and prioritizing safeguards in workflows where failures are most likely to occur.