A Meta-Summary of Challenges in Building Products with ML Components -- Collecting Experiences from 4758+ Practitioners

TL;DR

This paper synthesizes insights from 50 studies involving over 4758 practitioners to identify key challenges in integrating ML components into software products, aiming to guide future research and education.

Contribution

It provides a comprehensive meta-summary of industry challenges in building ML-enabled products by aggregating findings from numerous studies, offering a valuable resource for the research community.

Findings

Identified the most frequently reported challenges in ML product development.

Grouped and organized over 500 mentions of challenges from multiple studies.

Highlighted areas needing further research and educational focus.

Abstract

Incorporating machine learning (ML) components into software products raises new software-engineering challenges and exacerbates existing challenges. Many researchers have invested significant effort in understanding the challenges of industry practitioners working on building products with ML components, through interviews and surveys with practitioners. With the intention to aggregate and present their collective findings, we conduct a meta-summary study: We collect 50 relevant papers that together interacted with over 4758 practitioners using guidelines for systematic literature reviews. We then collected, grouped, and organized the over 500 mentions of challenges within those papers. We highlight the most commonly reported challenges and hope this meta-summary will be a useful resource for the research community to prioritize research and education in this field.