Theory and Practice of Quality Assurance for Machine Learning Systems An Experiment Driven Approach

TL;DR

This paper advocates an 'experiment first' approach for quality assurance in machine learning systems, emphasizing statistical control, continuous monitoring, and systematic experimentation throughout the system's lifecycle.

Contribution

It introduces a comprehensive framework combining business requirement quantification, statistical experimentation, and lifecycle control for ML system quality assurance.

Findings

Emphasizes early experiment design to reduce project failure risk.

Highlights importance of continuous statistical monitoring.

Provides detailed methodology for lifecycle quality control.

Abstract

The crafting of machine learning (ML) based systems requires statistical control throughout its life cycle. Careful quantification of business requirements and identification of key factors that impact the business requirements reduces the risk of a project failure. The quantification of business requirements results in the definition of random variables representing the system key performance indicators that need to be analyzed through statistical experiments. In addition, available data for training and experiment results impact the design of the system. Once the system is developed, it is tested and continually monitored to ensure it meets its business requirements. This is done through the continued application of statistical experiments to analyze and control the key performance indicators. This book teaches the art of crafting and developing ML based systems. It advocates an "experiment first" approach stressing the need to define statistical experiments from the beginning of the project life cycle. It also discusses in detail how to apply statistical control on the ML based system throughout its lifecycle.