Towards Guidelines for Assessing Qualities of Machine Learning Systems

TL;DR

This paper proposes a structured quality model for assessing machine learning systems, addressing the unique aspects and challenges in evaluating their quality compared to traditional software systems.

Contribution

It introduces a tailored quality model for ML systems, including evaluation objects, quality aspects, and metrics, based on an industrial use case.

Findings

A comprehensive quality model for ML systems is constructed.

The model enables objective specification and assessment of ML system qualities.

Future work includes deriving general guidelines for different ML system types.

Abstract

Nowadays, systems containing components based on machine learning (ML) methods are becoming more widespread. In order to ensure the intended behavior of a software system, there are standards that define necessary quality aspects of the system and its components (such as ISO/IEC 25010). Due to the different nature of ML, we have to adjust quality aspects or add additional ones (such as trustworthiness) and be very precise about which aspect is really relevant for which object of interest (such as completeness of training data), and how to objectively assess adherence to quality requirements. In this article, we present the construction of a quality model (i.e., evaluation objects, quality aspects, and metrics) for an ML system based on an industrial use case. This quality model enables practitioners to specify and assess quality requirements for such kinds of ML systems objectively. In the future, we want to learn how the term quality differs between different types of ML systems and come up with general guidelines for specifying and assessing qualities of ML systems.