Predicting Intermittent Job Failure Categories for Diagnosis Using Few-Shot Fine-Tuned Language Models

TL;DR

This paper introduces FlaXifyer, a few-shot learning method using language models to classify intermittent CI job failures, and LogSift, an interpretability technique to identify key log statements, improving diagnosis efficiency.

Contribution

The work presents a novel few-shot learning approach for failure classification and an interpretability method, addressing diagnosis of flaky CI job failures with minimal labeled data.

Findings

FlaXifyer achieves 84.3% Macro F1 with 12 examples per category.

LogSift reduces review effort by 74.4%.

Evaluation on 2,458 failures shows effective triage and diagnosis.

Abstract

In principle, Continuous Integration (CI) pipeline failures provide valuable feedback to developers on code-related errors. In practice, however, pipeline jobs often fail intermittently due to non-deterministic tests, network outages, infrastructure failures, resource exhaustion, and other reliability issues. These intermittent (flaky) job failures lead to substantial inefficiencies: wasted computational resources from repeated reruns and significant diagnosis time that distracts developers from core activities and often requires intervention from specialized teams. Prior work has proposed machine learning techniques to detect intermittent failures, but does not address the subsequent diagnosis challenge. To fill this gap, we introduce FlaXifyer, a few-shot learning approach for predicting intermittent job failure categories using pre-trained language models. FlaXifyer requires only job execution logs and achieves 84.3% Macro F1 and 92.0% Top-2 accuracy with just 12 labeled examples per category. We also propose LogSift, an interpretability technique that identifies influential log statements in under one second, reducing review effort by 74.4% while surfacing relevant failure information in 87% of cases. Evaluation on 2,458 job failures from TELUS demonstrates that FlaXifyer and LogSift enable effective automated triage, accelerate failure diagnosis, and pave the way towards the automated resolution of intermittent job failures.