Systematic Literature Review on Application of Machine Learning in Continuous Integration

TL;DR

This systematic review analyzes 22 years of literature on machine learning applications in Continuous Integration, detailing techniques, data sources, evaluation metrics, and relationships between ML models and CI phases.

Contribution

It provides a comprehensive synthesis of ML techniques, data handling, and evaluation methods used in CI, highlighting gaps and directions for future research.

Findings

Identified nine data source types and four data preparation steps.

Summarized five hyper-parameter tuning methods and fifteen evaluation metrics.

Mapped relationships between ML models, performance metrics, and CI phases.

Abstract

This research conducted a systematic review of the literature on machine learning (ML)-based methods in the context of Continuous Integration (CI) over the past 22 years. The study aimed to identify and describe the techniques used in ML-based solutions for CI and analyzed various aspects such as data engineering, feature engineering, hyper-parameter tuning, ML models, evaluation methods, and metrics. In this paper, we have depicted the phases of CI testing, the connection between them, and the employed techniques in training the ML method phases. We presented nine types of data sources and four taken steps in the selected studies for preparing the data. Also, we identified four feature types and nine subsets of data features through thematic analysis of the selected studies. Besides, five methods for selecting and tuning the hyper-parameters are shown. In addition, we summarised the evaluation methods used in the literature and identified fifteen different metrics. The most commonly used evaluation methods were found to be precision, recall, and F1-score, and we have also identified five methods for evaluating the performance of trained ML models. Finally, we have presented the relationship between ML model types, performance measurements, and CI phases. The study provides valuable insights for researchers and practitioners interested in ML-based methods in CI and emphasizes the need for further research in this area.