Efficient and Noise-Tolerant PAC Learning of Multiclass Linear Classifiers

TL;DR

This paper presents a computationally-efficient PAC learning algorithm for multiclass linear classifiers that tolerates malicious noise, using a mixture of bounded variance distributions and margin conditions.

Contribution

It introduces a novel algorithm combining cluster-based pruning and hinge loss minimization for noise-tolerant multiclass PAC learning.

Findings

The algorithm PAC learns with O(k^2(d log d + log k)) samples under constant noise.

It is strictly stronger than previous results even in the binary case.

The approach works under mixture distributions with margin conditions.

Abstract

Noise-tolerant PAC learning of linear models has been of central interests in machine learning community since the last century. In recent years, many computationally-efficient algorithms have been proposed for the problem of learning linear threshold functions under multiple noise models. Yet, when the problem is considered under multiclass learning settings, i.e. when the number of classes $k$ is at least $3$, it is unknown whether there exist computationally-efficient PAC learning algorithms when the data sets are maliciously corrupted. In this paper, we consider that the marginal distribution is a mixture of bounded variance distributions and the data sets satisfy a margin condition at the same time. We show that there exists a computationally-efficient algorithm that PAC learns multiclass linear classifiers $\{h_w:x\mapsto \arg\max_{y\in[k]}w_y\cdot x, x\in \mathbb{R}^d, w\in\mathbb{R}^{kd}\}$ using at most $O(k^2\cdot (d\log d+\log k))$ samples even under a constant rate of nasty noise. Our algorithm consists of two main ingredients: a cluster-based pruning scheme and a standard multiclass hinge loss minimization program. Even in the special case of binary setting, i.e. $k=2$, our result is strictly stronger than all prior works.