Using Large Language Models to Support Automation of Failure Management in CI/CD Pipelines: A Case Study in SAP HANA

TL;DR

This study evaluates the use of large language models to automate failure management in CI/CD pipelines for SAP HANA, demonstrating high accuracy in error localization and solution generation when provided with historical failure data.

Contribution

It shows that LLMs, combined with historical failure data, can effectively automate failure management tasks in industrial CI/CD pipelines, a novel application in this domain.

Findings

92.1% success in generating exact solutions

97.4% accuracy in error localization with domain knowledge

Historical failure data significantly improves system performance

Abstract

CI/CD pipeline failure management is time-consuming when performed manually. Automating this process is non-trivial because the information required for effective failure management is unstructured and cannot be automatically processed by traditional programs. With their ability to process unstructured data, large language models (LLMs) have shown promising results for automated failure management by previous work. Following these studies, we evaluated whether an LLM-based system could automate failure management in a CI/CD pipeline in the context of a large industrial software project, namely SAP HANA. We evaluated the ability of the LLM-based system to identify the error location and to propose exact solutions that contain no unnecessary actions. To support the LLM in generating exact solutions, we provided it with different types of domain knowledge, including pipeline information, failure management instructions, and data from historical failures. We conducted an ablation study to determine which type of domain knowledge contributed most to solution accuracy. The results show that data from historical failures contributed the most to the system's accuracy, enabling it to produce exact solutions in 92.1% of cases in our dataset. The system correctly identified the error location with 97.4% accuracy when provided with domain knowledge, compared to 84.2% accuracy without it. In conclusion, our findings indicate that LLMs, when provided with data from historical failures, represent a promising approach for automating CI/CD pipeline failure management.