VirnyFlow: A Design Space for Responsible Model Development

TL;DR

VirnyFlow is a comprehensive framework that enables responsible, customizable, and scalable machine learning pipeline development through multi-objective optimization and real-world constraint integration.

Contribution

VirnyFlow introduces a novel design space for responsible ML development, combining multi-objective optimization, customization, and scalability in a unified system.

Findings

Outperforms state-of-the-art AutoML in optimization quality

Demonstrates scalability across five real-world benchmarks

Provides a flexible, responsible alternative to black-box AutoML

Abstract

Developing machine learning (ML) models requires a deep understanding of real-world problems, which are inherently multi-objective. In this paper, we present VirnyFlow, the first design space for responsible model development, designed to assist data scientists in building ML pipelines that are tailored to the specific context of their problem. Unlike conventional AutoML frameworks, VirnyFlow enables users to define customized optimization criteria, perform comprehensive experimentation across pipeline stages, and iteratively refine models in alignment with real-world constraints. Our system integrates evaluation protocol definition, multi-objective Bayesian optimization, cost-aware multi-armed bandits, query optimization, and distributed parallelism into a unified architecture. We show that VirnyFlow significantly outperforms state-of-the-art AutoML systems in both optimization quality and scalability across five real-world benchmarks, offering a flexible, efficient, and responsible alternative to black-box automation in ML development.