Non-functional Requirements for Machine Learning: Understanding Current Use and Challenges in Industry

TL;DR

This paper explores how industry practitioners understand and manage non-functional requirements in machine learning systems, highlighting current practices, challenges, and the importance of NFRs like performance, transparency, and fairness.

Contribution

It provides empirical insights into industry practices and challenges in defining, measuring, and prioritizing NFRs for ML systems, guiding future requirements engineering efforts.

Findings

Examples of NFR identification and measurement in ML

Variations in NFR importance for ML systems

Challenges faced in industry when dealing with NFRs for ML

Abstract

Machine Learning (ML) is an application of Artificial Intelligence (AI) that uses big data to produce complex predictions and decision-making systems, which would be challenging to obtain otherwise. To ensure the success of ML-enabled systems, it is essential to be aware of certain qualities of ML solutions (performance, transparency, fairness), known from a Requirement Engineering (RE) perspective as non-functional requirements (NFRs). However, when systems involve ML, NFRs for traditional software may not apply in the same ways; some NFRs may become more prominent or less important; NFRs may be defined over the ML model, data, or the entire system; and NFRs for ML may be measured differently. In this work, we aim to understand the state-of-the-art and challenges of dealing with NFRs for ML in industry. We interviewed ten engineering practitioners working with NFRs and ML. We find examples of (1) the identification and measurement of NFRs for ML, (2) identification of more and less important NFRs for ML, and (3) the challenges associated with NFRs and ML in the industry. This knowledge paints a picture of how ML-related NFRs are treated in practice and helps to guide future RE for ML efforts.