A Survey of Human-in-the-loop for Machine Learning

TL;DR

This survey reviews human-in-the-loop machine learning, categorizing approaches into data improvement, interventional training, and system design, highlighting strengths, weaknesses, and future challenges across various domains.

Contribution

It provides a comprehensive classification and analysis of human-in-the-loop methods, offering insights into technical strengths, weaknesses, and open challenges in the field.

Findings

Classified human-in-the-loop approaches into three categories

Analyzed strengths and weaknesses of existing methods

Identified open challenges and future opportunities

Abstract

Human-in-the-loop aims to train an accurate prediction model with minimum cost by integrating human knowledge and experience. Humans can provide training data for machine learning applications and directly accomplish tasks that are hard for computers in the pipeline with the help of machine-based approaches. In this paper, we survey existing works on human-in-the-loop from a data perspective and classify them into three categories with a progressive relationship: (1) the work of improving model performance from data processing, (2) the work of improving model performance through interventional model training, and (3) the design of the system independent human-in-the-loop. Using the above categorization, we summarize major approaches in the field; along with their technical strengths/ weaknesses, we have simple classification and discussion in natural language processing, computer vision, and others. Besides, we provide some open challenges and opportunities. This survey intends to provide a high-level summarization for human-in-the-loop and motivates interested readers to consider approaches for designing effective human-in-the-loop solutions.