Design Patterns for Machine Learning Based Systems with Human-in-the-Loop

TL;DR

This paper presents a catalog of design patterns to effectively integrate human-in-the-loop strategies into machine learning systems, addressing challenges in reliability and human involvement.

Contribution

It introduces a comprehensive set of design patterns for human-in-the-loop ML systems, considering cost and retraining requirements.

Findings

Catalog of 10 design patterns for HiL ML systems

Guidelines for selecting and implementing patterns

Enhanced understanding of human integration in ML workflows

Abstract

The development and deployment of systems using supervised machine learning (ML) remain challenging: mainly due to the limited reliability of prediction models and the lack of knowledge on how to effectively integrate human intelligence into automated decision-making. Humans involvement in the ML process is a promising and powerful paradigm to overcome the limitations of pure automated predictions and improve the applicability of ML in practice. We compile a catalog of design patterns to guide developers select and implement suitable human-in-the-loop (HiL) solutions. Our catalog takes into consideration key requirements as the cost of human involvement and model retraining. It includes four training patterns, four deployment patterns, and two orthogonal cooperation patterns.