Scanflow: A multi-graph framework for Machine Learning workflow management, supervision, and debugging

TL;DR

Scanflow is a containerized multi-graph framework designed to manage, supervise, and debug machine learning workflows, enabling collaborative, modular, and flexible model monitoring and improvement in production environments.

Contribution

The paper introduces a novel containerized directed graph framework that supports end-to-end ML workflow management, supervision, and debugging with human and machine collaboration.

Findings

Effective detection of data drift in multiple datasets

Integration of human intervention improves model robustness

Promising accuracy results on benchmark datasets

Abstract

Machine Learning (ML) is more than just training models, the whole workflow must be considered. Once deployed, a ML model needs to be watched and constantly supervised and debugged to guarantee its validity and robustness in unexpected situations. Debugging in ML aims to identify (and address) the model weaknesses in not trivial contexts. Several techniques have been proposed to identify different types of model weaknesses, such as bias in classification, model decay, adversarial attacks, etc., yet there is not a generic framework that allows them to work in a collaborative, modular, portable, iterative way and, more importantly, flexible enough to allow both human- and machine-driven techniques. In this paper, we propose a novel containerized directed graph framework to support and accelerate end-to-end ML workflow management, supervision, and debugging. The framework allows defining and deploying ML workflows in containers, tracking their metadata, checking their behavior in production, and improving the models by using both learned and human-provided knowledge. We demonstrate these capabilities by integrating in the framework two hybrid systems to detect data drift distribution which identify the samples that are far from the latent space of the original distribution, ask for human intervention, and whether retrain the model or wrap it with a filter to remove the noise of corrupted data at inference time. We test these systems on MNIST-C, CIFAR-10-C, and FashionMNIST-C datasets, obtaining promising accuracy results with the help of human involvement.