Learning from Change: Predictive Models for Incident Prevention in a Regulated IT Environment

TL;DR

This paper introduces an interpretable machine learning approach to predict and prevent high-risk IT incidents caused by changes in a regulated banking environment, enhancing operational reliability.

Contribution

It presents a novel, explainable incident risk scoring model using SHAP values, outperforming rule-based methods in a real-world banking dataset.

Findings

LightGBM achieved the best predictive performance.

Aggregated team metrics improved model accuracy.

Data-driven models can outperform rule-based approaches in regulated settings.

Abstract

Effective IT change management is important for businesses that depend on software and services, particularly in highly regulated sectors such as finance, where operational reliability, auditability, and explainability are essential. A significant portion of IT incidents are caused by changes, making it important to identify high-risk changes before deployment. This study presents a predictive incident risk scoring approach at a large international bank. The approach supports engineers during the assessment and planning phases of change deployments by predicting the potential of inducing incidents. To satisfy regulatory constraints, we built the model with auditability and explainability in mind, applying SHAP values to provide feature-level insights and ensure decisions are traceable and transparent. Using a one-year real-world dataset, we compare the existing rule-based process with three machine learning models: HGBC, LightGBM, and XGBoost. LightGBM achieved the best performance, particularly when enriched with aggregated team metrics that capture organisational context. Our results show that data-driven, interpretable models can outperform rule-based approaches while meeting compliance needs, enabling proactive risk mitigation and more reliable IT operations.