Collaboration Challenges in Building ML-Enabled Systems: Communication, Documentation, Engineering, and Process

TL;DR

This paper identifies key collaboration challenges in building ML-enabled systems, highlighting issues in communication, documentation, engineering, and process, based on interviews with practitioners, and offers recommendations to address them.

Contribution

It provides an empirical analysis of collaboration challenges in ML system development and proposes practical recommendations for improvement.

Findings

Communication, documentation, engineering, and process are major challenges.

Teams face difficulties in requirements, data, and integration points.

Recommendations help improve collaboration and system quality.

Abstract

The introduction of machine learning (ML) components in software projects has created the need for software engineers to collaborate with data scientists and other specialists. While collaboration can always be challenging, ML introduces additional challenges with its exploratory model development process, additional skills and knowledge needed, difficulties testing ML systems, need for continuous evolution and monitoring, and non-traditional quality requirements such as fairness and explainability. Through interviews with 45 practitioners from 28 organizations, we identified key collaboration challenges that teams face when building and deploying ML systems into production. We report on common collaboration points in the development of production ML systems for requirements, data, and integration, as well as corresponding team patterns and challenges. We find that most of these challenges center around communication, documentation, engineering, and process and collect recommendations to address these challenges.