Can LLMs Write CI? A Study on Automatic Generation of GitHub Actions Configurations

TL;DR

This study evaluates the ability of six Large Language Models to generate GitHub Actions CI configurations from natural language, highlighting current limitations and introducing a new dataset for future research.

Contribution

It provides the first labeled dataset pairing descriptions with YAML configurations and assesses LLM performance in CI configuration generation.

Findings

Zero-shot prompting achieves up to 69% similarity with ground truth.

Only 3% of generated configurations are perfect matches.

Code-pretrained models slightly underperform general-purpose models.

Abstract

Continuous Integration (CI) services, such as GitHub Actions, require developers to write YAML-based configurations, which can be tedious and error-prone. Despite the increasing use of Large Language Models (LLMs) to automate software engineering tasks, their ability to generate CI configurations remains underexplored. This paper presents a preliminary study evaluating six LLMs for generating GitHub Actions configurations from natural language descriptions. We assess three general-purpose foundation models (GPT-4o, Llama, and Gemma) and three code-pretrained models (GPT-4.1, Code Llama, and CodeGemma). We also introduce the first labeled dataset of its kind, constructed from GitHub Actions documentation, pairing descriptions with corresponding best-practice YAML configurations. Zero-shot prompting achieves up to 69% similarity with the ground truth, with only 3% perfect matches. Code-pretrained models slightly underperform compared to general-purpose ones in YAML-based CI tasks, revealing LLM limitations for CI configuration generation. Analyzing GPT-4o outputs reveals issues like missing or renamed steps, misinterpreted descriptions, and unnecessary additions that may affect structural and contextual correctness, indicating a gap between generation quality and the precision required for executable CI configurations. Our research offers insights for improving LLM alignment with configuration languages and guiding future efforts on CI automation and tooling support.