On Proper Learnability between Average- and Worst-case Robustness

TL;DR

This paper explores proper learnability in adversarially robust PAC learning by examining relaxations of the robust loss, showing conditions under which VC classes are learnable and providing new generalization guarantees.

Contribution

It introduces new robust loss relaxations that enable proper PAC learning of VC classes and analyzes their learnability and generalization properties.

Findings

VC classes are properly PAC learnable under certain robust loss relaxations.

Finite VC dimension is insufficient for proper learning with some relaxations.

New generalization guarantees for adversarially robust empirical risk minimization.

Abstract

Recently, Montasser et al. [2019] showed that finite VC dimension is not sufficient for proper adversarially robust PAC learning. In light of this hardness, there is a growing effort to study what type of relaxations to the adversarially robust PAC learning setup can enable proper learnability. In this work, we initiate the study of proper learning under relaxations of the worst-case robust loss. We give a family of robust loss relaxations under which VC classes are properly PAC learnable with sample complexity close to what one would require in the standard PAC learning setup. On the other hand, we show that for an existing and natural relaxation of the worst-case robust loss, finite VC dimension is not sufficient for proper learning. Lastly, we give new generalization guarantees for the adversarially robust empirical risk minimizer.