Intelligent resource prediction for SAP HANA continuous integration build workloads

TL;DR

This paper presents an intelligent resource prediction system using machine learning to optimize memory allocation in large-scale CI pipelines, significantly reducing over-provisioning and under-allocation.

Contribution

It introduces a LightGBM-XGBoost quantile regression ensemble for memory prediction, integrated into production CI pipelines to improve resource efficiency.

Findings

Achieved approximately 36GB memory savings per build.

Reduced under-allocation rates to below 0.3%.

Demonstrated effectiveness on over 300,000 build executions.

Abstract

Large enterprises often operate extensive Continuous Integration (CI) pipelines on large, heterogeneous compute clusters, where conservative, statically defined resource requirements are used to ensure build reliability. This practice leads to substantial system memory over-allocation, reduced cluster utilization, and increased operational costs. In this paper, we motivate the need for intelligent resource prediction by analyzing over 300,000 historical build executions from a production CI environment with more than one thousand compute nodes. Our analysis shows that, on average, more than 60% of allocated system memory remains unused. We then compare multiple machine learning approaches for predicting build task memory usage, including classification-based methods and regression-based quantile prediction. Our final solution employs a LightGBM-XGBoost quantile regression ensemble optimized to minimize under-allocation while reducing over-provisioning. We integrate this solution into the production CI pipeline via a microservice-based orchestration layer, achieving average memory savings of approximately 36GB per build and reducing under-allocation rates to below 0.3% without negatively impacting build execution times.