Refining Labeling Functions with Limited Labeled Data

TL;DR

This paper introduces techniques to improve labeling functions in programmatic weak supervision by minimally repairing them using a small labeled dataset, leading to more accurate data labeling.

Contribution

We develop novel methods for repairing labeling functions with limited labeled data, ensuring higher accuracy and evidence support for correct labels.

Findings

Improved LF quality with small labeled datasets

Minimal changes to LFs can significantly enhance accuracy

Effective refinement of conditional rule-based LFs

Abstract

Programmatic weak supervision (PWS) significantly reduces human effort for labeling data by combining the outputs of user-provided labeling functions (LFs) on unlabeled datapoints. However, the quality of the generated labels depends directly on the accuracy of the LFs. In this work, we study the problem of fixing LFs based on a small set of labeled examples. Towards this goal, we develop novel techniques for repairing a set of LFs by minimally changing their results on the labeled examples such that the fixed LFs ensure that (i) there is sufficient evidence for the correct label of each labeled datapoint and (ii) the accuracy of each repaired LF is sufficiently high. We model LFs as conditional rules which enables us to refine them, i.e., to selectively change their output for some inputs. We demonstrate experimentally that our system improves the quality of LFs based on surprisingly small sets of labeled datapoints.