Maturity Framework for Enhancing Machine Learning Quality

TL;DR

This paper introduces a comprehensive maturity framework and open-source assessment method to improve the quality, reliability, and governance of machine learning systems, validated through empirical evidence and industry application.

Contribution

It presents a novel structured maturity framework and assessment approach for ML quality, addressing gaps in governance and providing practical validation from real-world deployment.

Findings

Quality improvement trends observed in empirical data

Enhanced ML governance through structured assessment

Positive business outcomes from framework adoption

Abstract

With the rapid integration of Machine Learning (ML) in business applications and processes, it is crucial to ensure the quality, reliability and reproducibility of such systems. We suggest a methodical approach towards ML system quality assessment and introduce a structured Maturity framework for governance of ML. We emphasize the importance of quality in ML and the need for rigorous assessment, driven by issues in ML governance and gaps in existing frameworks. Our primary contribution is a comprehensive open-sourced quality assessment method, validated with empirical evidence, accompanied by a systematic maturity framework tailored to ML systems. Drawing from applied experience at Booking.com, we discuss challenges and lessons learned during large-scale adoption within organizations. The study presents empirical findings, highlighting quality improvement trends and showcasing business outcomes. The maturity framework for ML systems, aims to become a valuable resource to reshape industry standards and enable a structural approach to improve ML maturity in any organization.