The Product Beyond the Model -- An Empirical Study of Repositories of Open-Source ML Products

TL;DR

This study provides an empirical analysis of 262 open-source ML products, revealing development practices, architectural choices, and gaps in industry best practices, with implications for research, education, and tool development.

Contribution

It introduces a new dataset of open-source ML products and offers a comprehensive analysis of their development practices and architecture, addressing a gap in empirical research.

Findings

Most ML products resemble startup-style development.

Limited involvement of data scientists in open-source ML projects.

Low modularity between ML and non-ML code.

Abstract

Machine learning (ML) components are increasingly incorporated into software products for end-users, but developers face challenges in transitioning from ML prototypes to products. Academics have limited access to the source of commercial ML products, hindering research progress to address these challenges. In this study, first and foremost, we contribute a dataset of 262 open-source ML products for end users (not just models), identified among more than half a million ML-related projects on GitHub. Then, we qualitatively and quantitatively analyze 30 open-source ML products to answer six broad research questions about development practices and system architecture. We find that the majority of the ML products in our sample represent more startup-style development than reported in past interview studies. We report 21 findings, including limited involvement of data scientists in many open-source ML products, unusually low modularity between ML and non-ML code, diverse architectural choices on incorporating models into products, and limited prevalence of industry best practices such as model testing, pipeline automation, and monitoring. Additionally, we discuss seven implications of this study on research, development, and education, including the need for tools to assist teams without data scientists, education opportunities, and open-source-specific research for privacy-preserving telemetry.