TagRuler: Interactive Tool for Span-Level Data Programming by Demonstration

TL;DR

TagRuler is an interactive tool that enables non-programmers to create span-level data labeling functions efficiently, improving annotation quality and reducing time for NLP tasks through demonstration-based data programming.

Contribution

This work extends the Data Programming by Demonstration framework to span-level NLP annotation, introducing a user-friendly tool that enhances annotation accuracy without requiring programming skills.

Findings

Higher F1 scores achieved with TagRuler compared to manual labeling

Effective span-level annotation facilitated without programming

Empirical validation across multiple NLP tasks

Abstract

Despite rapid developments in the field of machine learning research, collecting high-quality labels for supervised learning remains a bottleneck for many applications. This difficulty is exacerbated by the fact that state-of-the-art models for NLP tasks are becoming deeper and more complex, often increasing the amount of training data required even for fine-tuning. Weak supervision methods, including data programming, address this problem and reduce the cost of label collection by using noisy label sources for supervision. However, until recently, data programming was only accessible to users who knew how to program. To bridge this gap, the Data Programming by Demonstration framework was proposed to facilitate the automatic creation of labeling functions based on a few examples labeled by a domain expert. This framework has proven successful for generating high-accuracy labeling models for document classification. In this work, we extend the DPBD framework to span-level annotation tasks, arguably one of the most time-consuming NLP labeling tasks. We built a novel tool, TagRuler, that makes it easy for annotators to build span-level labeling functions without programming and encourages them to explore trade-offs between different labeling models and active learning strategies. We empirically demonstrated that an annotator could achieve a higher F1 score using the proposed tool compared to manual labeling for different span-level annotation tasks.