Risk-Consistent Multiclass Learning from Random Label-Subset Membership Queries

TL;DR

This paper introduces a new multiclass learning framework using random label-subset queries, providing unbiased risk estimators and theoretical guarantees, addressing practical limitations of label acquisition.

Contribution

It systematically characterizes weak supervision from label-subset queries, deriving unbiased and corrected risk estimators with theoretical analysis and experimental validation.

Findings

Unbiased risk estimator enables effective learning from label-subset queries.

Corrected risk estimators improve stability and reduce overfitting.

Experimental results confirm the feasibility of direct query-response learning.

Abstract

Obtaining accurate class labels is often costly or unreliable, and may also be limited by privacy or other practical conditions. Compared with asking an annotator to provide the exact class, it is often easier to ask whether the true label belongs to a certain label subset. This query-response form defines a distinct weak-supervision mechanism: weak supervision information is generated through feedback on a label subset. Although weakly supervised learning has studied many learning frameworks, most existing work starts from established weak label objects. A systematic characterization is still lacking for weakly supervised learning generated directly by such query response observations. This paper proposes a multiclass learn ing framework under random label-subset queries. We model the data-generating distribution of query-response observations and derive an unbiased estimator of the target risk under the empirical risk minimization (ERM) framework. To address negative empirical risk and the associated overfitting problem, we introduce corrected risk estimators based on non-negative and absolute-value corrections. Theoretical analysis establishes a conditional generalization and excess-risk bound for the unbiased estimator, and a bias-and-consistency result for the corrected risk estimator. Experiments under the matched random-query mechanism demonstrate the feasibility of direct query-response learning and the stabilization effect of risk correction.