Convergence Behavior of an Adversarial Weak Supervision Method

TL;DR

This paper analyzes the convergence properties of an adversarial weak supervision method, providing theoretical insights and experimental validation, and compares it to probabilistic approaches which may lack consistency.

Contribution

It offers a comprehensive statistical analysis of the adversarial approach under log-loss, including solution characterization, consistency, and convergence rates, highlighting differences from probabilistic methods.

Findings

Adversarial approach solutions relate to logistic regression.

The adversarial method is shown to be consistent.

Probabilistic methods can fail to be consistent.

Abstract

Labeling data via rules-of-thumb and minimal label supervision is central to Weak Supervision, a paradigm subsuming subareas of machine learning such as crowdsourced learning and semi-supervised ensemble learning. By using this labeled data to train modern machine learning methods, the cost of acquiring large amounts of hand labeled data can be ameliorated. Approaches to combining the rules-of-thumb falls into two camps, reflecting different ideologies of statistical estimation. The most common approach, exemplified by the Dawid-Skene model, is based on probabilistic modeling. The other, developed in the work of Balsubramani-Freund and others, is adversarial and game-theoretic. We provide a variety of statistical results for the adversarial approach under log-loss: we characterize the form of the solution, relate it to logistic regression, demonstrate consistency, and give rates of convergence. On the other hand, we find that probabilistic approaches for the same model class can fail to be consistent. Experimental results are provided to corroborate the theoretical results.