Attribute-Efficient PAC Learning of Sparse Halfspaces with Constant Malicious Noise Rate

TL;DR

This paper presents a new attribute-efficient PAC learning algorithm for sparse halfspaces that is robust to a constant rate of malicious noise, using hinge loss minimization with novel gradient analysis.

Contribution

It introduces the first attribute-efficient PAC learning method for sparse halfspaces resilient to malicious noise with a new gradient analysis approach.

Findings

Achieves learning with polynomial sample complexity in sparsity and log dimension.

Handles a constant malicious noise rate effectively.

Employs a modified hinge loss minimization with robust gradient analysis.

Abstract

Attribute-efficient PAC learning of sparse halfspaces has been a fundamental problem in machine learning theory. In recent years, machine learning algorithms are faced with prevalent data corruptions or even malicious attacks. It is of central interest to design computationally-efficient algorithms that are robust to malicious corruptions. In this paper, we consider that there exists a constant amount of malicious noise in the data and the goal is to learn an underlying $s$-sparse halfspace $w^* \in \mathbb{R}^d$ with $\text{poly}(s,\log d)$ samples. Specifically, we follow a recent line of works and assume that the underlying distribution satisfies a certain concentration condition and a margin condition at the same time. Under such conditions, we show that attribute-efficiency can be achieved with simple variants to existing hinge loss minimization programs. Our key contribution includes: 1) an attribute-efficient PAC learning algorithm that works under a constant malicious noise rate; 2) a new gradient analysis that carefully handles the sparsity admitted constraints in hinge loss minimization program.