A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges

TL;DR

This paper systematically reviews the current state of software engineering research for machine learning systems, highlighting the challenges, lack of mature tools, and the need for further empirical studies in industrial contexts.

Contribution

It provides a comprehensive analysis of existing SE approaches for ML systems, identifying gaps and emphasizing the importance of empirical research and tool development.

Findings

Testing is the most researched SE aspect for ML systems.

No mature tools or techniques currently exist for most SE aspects.

Challenges of ML systems engineering are mainly identified through surveys and interviews.

Abstract

Context: Advancements in machine learning (ML) lead to a shift from the traditional view of software development, where algorithms are hard-coded by humans, to ML systems materialized through learning from data. Therefore, we need to revisit our ways of developing software systems and consider the particularities required by these new types of systems. Objective: The purpose of this study is to systematically identify, analyze, summarize, and synthesize the current state of software engineering (SE) research for engineering ML systems. Method: I performed a systematic literature review (SLR). I systematically selected a pool of 141 studies from SE venues and then conducted a quantitative and qualitative analysis using the data extracted from these studies. Results: The non-deterministic nature of ML systems complicates all SE aspects of engineering ML systems. Despite increasing interest from 2018 onwards, the results reveal that none of the SE aspects have a mature set of tools and techniques. Testing is by far the most popular area among researchers. Even for testing ML systems, engineers have only some tool prototypes and solution proposals with weak experimental proof. Many of the challenges of ML systems engineering were identified through surveys and interviews. Researchers should conduct experiments and case studies, ideally in industrial environments, to further understand these challenges and propose solutions. Conclusion: The results may benefit (1) practitioners in foreseeing the challenges of ML systems engineering; (2) researchers and academicians in identifying potential research questions; and (3) educators in designing or updating SE courses to cover ML systems engineering.