OneLabeler: A Flexible System for Building Data Labeling Tools

TL;DR

OneLabeler is a flexible, visual programming system that simplifies the creation of diverse data labeling tools, reducing development time and effort for supervised machine learning datasets.

Contribution

It introduces a conceptual framework and a visual programming system for easily building customizable data labeling tools, enhancing flexibility and efficiency.

Findings

Supports configuration of human, machine, or mixed modules

Demonstrated with ten example labeling tools

User study shows improved development efficiency

Abstract

Labeled datasets are essential for supervised machine learning. Various data labeling tools have been built to collect labels in different usage scenarios. However, developing labeling tools is time-consuming, costly, and expertise-demanding on software development. In this paper, we propose a conceptual framework for data labeling and OneLabeler based on the conceptual framework to support easy building of labeling tools for diverse usage scenarios. The framework consists of common modules and states in labeling tools summarized through coding of existing tools. OneLabeler supports configuration and composition of common software modules through visual programming to build data labeling tools. A module can be a human, machine, or mixed computation procedure in data labeling. We demonstrate the expressiveness and utility of the system through ten example labeling tools built with OneLabeler. A user study with developers provides evidence that OneLabeler supports efficient building of diverse data labeling tools.