On the Interaction between Software Engineers and Data Scientists when building Machine Learning-Enabled Systems

TL;DR

This paper explores the collaboration challenges between software engineers and data scientists in ML projects through a case study, highlighting issues and proposing solutions to improve teamwork and system documentation.

Contribution

It provides an empirical analysis of role interactions in ML projects, identifying key collaboration challenges and suggesting practical improvements.

Findings

Differences in technical expertise hinder collaboration

Unclear role definitions create confusion

Lack of documentation impairs system specification

Abstract

In recent years, Machine Learning (ML) components have been increasingly integrated into the core systems of organizations. Engineering such systems presents various challenges from both a theoretical and practical perspective. One of the key challenges is the effective interaction between actors with different backgrounds who need to work closely together, such as software engineers and data scientists. This paper presents an exploratory case study to understand the current interaction and collaboration dynamics between these roles in ML projects. We conducted semi-structured interviews with four practitioners with experience in software engineering and data science of a large ML-enabled system project and analyzed the data using reflexive thematic analysis. Our findings reveal several challenges that can hinder collaboration between software engineers and data scientists, including differences in technical expertise, unclear definitions of each role's duties, and the lack of documents that support the specification of the ML-enabled system. We also indicate potential solutions to address these challenges, such as fostering a collaborative culture, encouraging team communication, and producing concise system documentation. This study contributes to understanding the complex dynamics between software engineers and data scientists in ML projects and provides insights for improving collaboration and communication in this context. We encourage future studies investigating this interaction in other projects.