Efficient Detection of Intermittent Job Failures Using Few-Shot Learning

TL;DR

This paper presents a few-shot learning approach for detecting intermittent job failures in CI pipelines, outperforming existing heuristics and models, with high accuracy using minimal labeled data.

Contribution

It introduces a novel few-shot learning method that effectively detects intermittent failures with limited labeled examples, improving over state-of-the-art heuristics and models.

Findings

Achieves 70-88% F1-score with only 12 labeled examples per project.

Outperforms the state-of-the-art heuristic, which scores 34-52% F1.

Mislabeled 32% of intermittent failures as regular in prior datasets.

Abstract

One of the main challenges developers face in the use of continuous integration (CI) and deployment pipelines is the occurrence of intermittent job failures, which result from unexpected non-deterministic issues (e.g., flaky tests or infrastructure problems) rather than regular code-related errors such as bugs. Prior studies developed machine learning (ML) models trained on large datasets of job logs to classify job failures as either intermittent or regular. As an alternative to costly manual labeling of large datasets, the state-of-the-art (SOTA) approach leveraged a heuristic based on non-deterministic job reruns. However, this method mislabels intermittent job failures as regular in contexts where rerunning suspicious job failures is not an explicit policy, and therefore limits the SOTA's performance in practice. In fact, our manual analysis of 2,125 job failures from 5 industrial and 1 open-source projects reveals that, on average, 32% of intermittent job failures are mislabeled as regular. To address these limitations, this paper introduces a novel approach to intermittent job failure detection using few-shot learning (FSL). Specifically, we fine-tune a small language model using a few number of manually labeled log examples to generate rich embeddings, which are then used to train an ML classifier. Our FSL-based approach achieves 70-88% F1-score with only 12 shots in all projects, outperforming the SOTA, which proved ineffective (34-52% F1-score) in 4 projects. Overall, this study underlines the importance of data quality over quantity and provides a more efficient and practical framework for the detection of intermittent job failures in organizations.