How are MLOps Frameworks Used in Open Source Projects? An Empirical Characterization

TL;DR

This study empirically analyzes how open-source MLOps frameworks are used in practice, revealing limited out-of-the-box use, custom API integrations, and common user requests for core feature enhancements and better CI/CD support.

Contribution

It provides the first empirical characterization of real-world usage and user needs for popular open-source MLOps frameworks, highlighting gaps between features and user demands.

Findings

Frameworks are rarely used out-of-the-box

Developers prefer custom API integrations

Users request core feature enhancements and better CI/CD support

Abstract

Machine Learning (ML) Operations (MLOps) frameworks have been conceived to support developers and AI engineers in managing the lifecycle of their ML models. While such frameworks provide a wide range of features, developers may leverage only a subset of them, while missing some highly desired features. This paper investigates the practical use and desired feature enhancements of eight popular open-source MLOps frameworks. Specifically, we analyze their usage by dependent projects on GitHub, examining how they invoke the frameworks' APIs and commands. Then, we qualitatively analyze feature requests and enhancements mined from the frameworks' issue trackers, relating these desired improvements to the previously identified usage features. Results indicate that MLOps frameworks are rarely used out-of-the-box and are infrequently integrated into GitHub Workflows, but rather, developers use their APIs to implement custom functionality in their projects. Used features concern core ML phases and whole infrastructure governance, sometimes leveraging multiple frameworks with complementary features. The mapping with feature requests highlights that users mainly ask for enhancements to core features of the frameworks, but also better API exposure and CI/CD integration.