Exploringand Unleashing the Power of Large Language Models in CI/CD Configuration Translation

TL;DR

This paper investigates the use of large language models for translating CI configurations during platform migration, analyzing effort, issues, and enhancement strategies to improve translation accuracy and success rates.

Contribution

It provides a comprehensive study on LLM-based CI configuration translation, including empirical analysis of migration effort, issue categorization, and effective prompting strategies.

Findings

Developers read 38 lines of Travis configs and write 58 lines of GitHub Actions configs.

Identified 1,121 issues in translations, mainly logic inconsistencies and platform discrepancies.

Combining guideline prompts with iterative refinement improves build success rate to 75.5%.

Abstract

Continuous Integration (CI) is a cornerstone of modern collaborative software development, and numerous CI platforms are available. Differences in maintenance overhead, reliability, and integration depth with code-hosting platforms make migration between CI platforms a common practice. A central step in migration is translating CI configurations, which is challenging due to the intrinsic complexity of CI configurations and the need to understand semantic differences and relationships across CI platforms. With the advent of large language models (LLMs), recent advances in software engineering highlight their potential for CI configuration translation. In this paper, we present a study on LLM-based CI configuration translation, focusing on the migration from Travis CI to GitHub Actions. First, using 811 migration records, we quantify the effort involved and find that developers read an average of 38 lines of Travis configuration and write 58 lines of GitHub Actions configuration, with nearly half of the migrations requiring multiple commits. We further analyze translations produced by each of the four LLMs and identify 1,121 issues grouped into four categories: logic inconsistencies (38%), platform discrepancies (32%), environment errors (25%), and syntax errors (5%). Finally, we evaluate three enhancement strategies and show that combining guideline-based prompting with iterative refinement achieves the best performance, reaching a Build Success Rate of 75.5%-nearly a threefold improvement over GPT-4o with a basic prompt.