A Large-Scale Study of Model Integration in ML-Enabled Software Systems

TL;DR

This large-scale study investigates the characteristics, reuse practices, and integration patterns of nearly 3,000 open-source ML-enabled software systems to improve understanding and engineering practices for such systems.

Contribution

It provides the first comprehensive empirical analysis of real-world ML-enabled systems, revealing common practices and architectural patterns.

Findings

Traditional code dominates ML-enabled systems

ML model reuse often involves code duplication or pre-trained models

Identified various ML integration patterns

Abstract

The rise of machine learning (ML) and its integration into software systems has drastically changed development practices. While software engineering traditionally focused on manually created code artifacts with dedicated processes and architectures, ML-enabled systems require additional data-science methods and tools to create ML artifacts -- especially ML models and training data. However, integrating models into systems, and managing the many different artifacts involved, is far from trivial. ML-enabled systems can easily have multiple ML models that interact with each other and with traditional code in intricate ways. Unfortunately, while challenges and practices of building ML-enabled systems have been studied, little is known about the characteristics of real-world ML-enabled systems beyond isolated examples. Improving engineering processes and architectures for ML-enabled systems requires improving the empirical understanding of these systems. We present a large-scale study of 2,928 open-source ML-enabled software systems. We classified and analyzed them to determine system characteristics, model and code reuse practices, and architectural aspects of integrating ML models. Our findings show that these systems still mainly consist of traditional source code, and that ML model reuse through code duplication or pre-trained models is common. We also identified different ML integration patterns and related implementation practices. We hope that our results help improve practices for integrating ML models, bringing data science and software engineering closer together.