LogSage: An LLM-Based Framework for CI/CD Failure Detection and Remediation with Industrial Validation

TL;DR

LogSage is an innovative LLM-based framework that automates root cause analysis and remediation of CI/CD failures, significantly improving accuracy and efficiency in industrial settings.

Contribution

It introduces the first end-to-end LLM-powered system for CI/CD failure diagnosis and fixes, combining log preprocessing, structured prompting, retrieval-augmented generation, and automation.

Findings

Achieves over 98% precision in failure diagnosis on a new benchmark.

Improves F1 score by more than 38 percentage points over recent baselines.

Processes over 1 million executions with over 80% end-to-end accuracy in industry.

Abstract

Continuous Integration and Deployment (CI/CD) pipelines are critical to modern software engineering, yet diagnosing and resolving their failures remains complex and labor-intensive. We present LogSage, the first end-to-end LLM-powered framework for root cause analysis (RCA) and automated remediation of CI/CD failures. LogSage employs a token-efficient log preprocessing pipeline to filter noise and extract critical errors, then performs structured diagnostic prompting for accurate RCA. For solution generation, it leverages retrieval-augmented generation (RAG) to reuse historical fixes and invokes automation fixes via LLM tool-calling. On a newly curated benchmark of 367 GitHub CI/CD failures, LogSage achieves over 98\% precision, near-perfect recall, and an F1 improvement of more than 38\% points in the RCA stage, compared with recent LLM-based baselines. In a year-long industrial deployment at ByteDance, it processed over 1.07M executions, with end-to-end precision exceeding 80\%. These results demonstrate that LogSage provides a scalable and practical solution for automating CI/CD failure management in real-world DevOps workflows.