Data Programming: Creating Large Training Sets, Quickly

TL;DR

Data programming enables rapid creation of large training datasets by using weak supervision through labeling functions, which are then denoised with a generative model, significantly improving model performance and accessibility.

Contribution

The paper introduces a novel paradigm for programmatically creating training data using labeling functions and a generative model to denoise labels, with theoretical guarantees and practical improvements.

Findings

Achieved a new winning score on the TAC-KBP challenge.

Improved LSTM performance by nearly 6 F1 points using data programming.

Demonstrated ease of use for non-experts in creating training data.

Abstract

Large labeled training sets are the critical building blocks of supervised learning methods and are key enablers of deep learning techniques. For some applications, creating labeled training sets is the most time-consuming and expensive part of applying machine learning. We therefore propose a paradigm for the programmatic creation of training sets called data programming in which users express weak supervision strategies or domain heuristics as labeling functions, which are programs that label subsets of the data, but that are noisy and may conflict. We show that by explicitly representing this training set labeling process as a generative model, we can "denoise" the generated training set, and establish theoretically that we can recover the parameters of these generative models in a handful of settings. We then show how to modify a discriminative loss function to make it noise-aware, and demonstrate our method over a range of discriminative models including logistic regression and LSTMs. Experimentally, on the 2014 TAC-KBP Slot Filling challenge, we show that data programming would have led to a new winning score, and also show that applying data programming to an LSTM model leads to a TAC-KBP score almost 6 F1 points over a state-of-the-art LSTM baseline (and into second place in the competition). Additionally, in initial user studies we observed that data programming may be an easier way for non-experts to create machine learning models when training data is limited or unavailable.