Training Classifiers with Natural Language Explanations

TL;DR

BabbleLabble enables training classifiers efficiently by leveraging natural language explanations to generate labeling functions, significantly reducing labeling effort while maintaining high accuracy.

Contribution

This work introduces BabbleLabble, a novel framework that converts natural language explanations into labeling functions for faster classifier training.

Findings

Users can train classifiers 5-100x faster with explanations.

A simple rule-based parser is sufficient for generating effective labeling functions.

Classifiers achieve comparable F1 scores using explanations versus traditional labels.

Abstract

Training accurate classifiers requires many labels, but each label provides only limited information (one bit for binary classification). In this work, we propose BabbleLabble, a framework for training classifiers in which an annotator provides a natural language explanation for each labeling decision. A semantic parser converts these explanations into programmatic labeling functions that generate noisy labels for an arbitrary amount of unlabeled data, which is used to train a classifier. On three relation extraction tasks, we find that users are able to train classifiers with comparable F1 scores from 5-100$\times$ faster by providing explanations instead of just labels. Furthermore, given the inherent imperfection of labeling functions, we find that a simple rule-based semantic parser suffices.