Interactive Machine Teaching by Labeling Rules and Instances

TL;DR

This paper introduces INTERVAL, an interactive framework that combines rule extraction and instance labeling to efficiently utilize expert input in weakly supervised learning, outperforming existing methods.

Contribution

The paper proposes a novel interactive learning framework, INTERVAL, that leverages language models and expert feedback on rules and instances to improve weak supervision efficiency.

Findings

INTERVAL outperforms state-of-the-art weakly supervised methods by 7% in F1 score.

It requires fewer expert queries—around 10—to reach high performance levels.

Rule precision is more critical than coverage in creating effective supervision.

Abstract

Weakly supervised learning aims to reduce the cost of labeling data by using expert-designed labeling rules. However, existing methods require experts to design effective rules in a single shot, which is difficult in the absence of proper guidance and tooling. Therefore, it is still an open question whether experts should spend their limited time writing rules or instead providing instance labels via active learning. In this paper, we investigate how to exploit an expert's limited time to create effective supervision. First, to develop practical guidelines for rule creation, we conduct an exploratory analysis of diverse collections of existing expert-designed rules and find that rule precision is more important than coverage across datasets. Second, we compare rule creation to individual instance labeling via active learning and demonstrate the importance of both across 6 datasets. Third, we propose an interactive learning framework, INTERVAL, that achieves efficiency by automatically extracting candidate rules based on rich patterns (e.g., by prompting a language model), and effectiveness by soliciting expert feedback on both candidate rules and individual instances. Across 6 datasets, INTERVAL outperforms state-of-the-art weakly supervised approaches by 7% in F1. Furthermore, it requires as few as 10 queries for expert feedback to reach F1 values that existing active learning methods cannot match even with 100 queries.