WALNUT: A Benchmark on Semi-weakly Supervised Learning for Natural Language Understanding

TL;DR

WALNUT is a comprehensive benchmark designed to evaluate semi-weakly supervised learning methods across diverse NLU tasks, facilitating systematic comparison and advancing research in low-resource language understanding.

Contribution

This paper introduces WALNUT, the first unified benchmark with real-world weak labels for multiple NLU tasks, enabling systematic evaluation of weak supervision techniques.

Findings

Weak supervision improves low-resource NLU performance.

Different weak sources have varying impacts across tasks.

Baseline models show significant gains with weak labels.

Abstract

Building machine learning models for natural language understanding (NLU) tasks relies heavily on labeled data. Weak supervision has been proven valuable when large amount of labeled data is unavailable or expensive to obtain. Existing works studying weak supervision for NLU either mostly focus on a specific task or simulate weak supervision signals from ground-truth labels. It is thus hard to compare different approaches and evaluate the benefit of weak supervision without access to a unified and systematic benchmark with diverse tasks and real-world weak labeling rules. In this paper, we propose such a benchmark, named WALNUT (semi-WeAkly supervised Learning for Natural language Understanding Testbed), to advocate and facilitate research on weak supervision for NLU. WALNUT consists of NLU tasks with different types, including document-level and token-level prediction tasks. WALNUT is the first semi-weakly supervised learning benchmark for NLU, where each task contains weak labels generated by multiple real-world weak sources, together with a small set of clean labels. We conduct baseline evaluations on WALNUT to systematically evaluate the effectiveness of various weak supervision methods and model architectures. Our results demonstrate the benefit of weak supervision for low-resource NLU tasks and highlight interesting patterns across tasks. We expect WALNUT to stimulate further research on methodologies to leverage weak supervision more effectively. The benchmark and code for baselines are available at \url{aka.ms/walnut_benchmark}.