Making a Pipeline Production-Ready: Challenges and Lessons Learned in the Healthcare Domain

TL;DR

This paper discusses the challenges and lessons learned in deploying a healthcare machine learning pipeline into production, highlighting architectural evolution and improvements in software quality attributes.

Contribution

It presents an experience report on transitioning from a proof-of-concept to a microservices architecture for a healthcare ML system, emphasizing practical insights.

Findings

Each architectural version improved extensibility, maintainability, robustness, and resiliency.

Transitioning to microservices enhanced system quality attributes.

Lessons learned can guide production deployment of ML pipelines in healthcare.

Abstract

Deploying a Machine Learning (ML) training pipeline into production requires good software engineering practices. Unfortunately, the typical data science workflow often leads to code that lacks critical software quality attributes. This experience report investigates this problem in SPIRA, a project whose goal is to create an ML-Enabled System (MLES) to pre-diagnose insufficiency respiratory via speech analysis. This paper presents an overview of the architecture of the MLES, then compares three versions of its Continuous Training subsystem: from a proof of concept Big Ball of Mud (v1), to a design pattern-based Modular Monolith (v2), to a test-driven set of Microservices (v3) Each version improved its overall extensibility, maintainability, robustness, and resiliency. The paper shares challenges and lessons learned in this process, offering insights for researchers and practitioners seeking to productionize their pipelines.