Learning Sparse Low-Threshold Linear Classifiers

TL;DR

This paper introduces an efficient method for learning non-negative linear classifiers with bounded norm and fixed threshold, achieving optimal rates that outperform traditional uniform convergence bounds.

Contribution

The paper presents the first efficient algorithms with optimal learning rates for sparse low-threshold linear classifiers under hinge-loss.

Findings

Efficient online and batch algorithms achieve the optimal rate of learning.

The proposed rates are tighter than traditional uniform convergence bounds.

The work generalizes learning to k-monotone disjunctions.

Abstract

We consider the problem of learning a non-negative linear classifier with a $1$-norm of at most $k$, and a fixed threshold, under the hinge-loss. This problem generalizes the problem of learning a $k$-monotone disjunction. We prove that we can learn efficiently in this setting, at a rate which is linear in both $k$ and the size of the threshold, and that this is the best possible rate. We provide an efficient online learning algorithm that achieves the optimal rate, and show that in the batch case, empirical risk minimization achieves this rate as well. The rates we show are tighter than the uniform convergence rate, which grows with $k^2$.