Beyond Perturbations: Learning Guarantees with Arbitrary Adversarial Test Examples

TL;DR

This paper introduces a transductive learning algorithm that guarantees low error and rejection rates on arbitrary, potentially adversarial test examples, even without assuming small perturbations from the training distribution.

Contribution

It provides the first nontrivial guarantees for learning with arbitrary train and test distributions using selective transductive learning for classes of bounded VC dimension.

Findings

Guarantees low test error and rejection rates for any test distribution.

Efficient algorithm given an ERM for the class.

Applicable to adversarially chosen test examples.

Abstract

We present a transductive learning algorithm that takes as input training examples from a distribution $P$ and arbitrary (unlabeled) test examples, possibly chosen by an adversary. This is unlike prior work that assumes that test examples are small perturbations of $P$. Our algorithm outputs a selective classifier, which abstains from predicting on some examples. By considering selective transductive learning, we give the first nontrivial guarantees for learning classes of bounded VC dimension with arbitrary train and test distributions---no prior guarantees were known even for simple classes of functions such as intervals on the line. In particular, for any function in a class $C$ of bounded VC dimension, we guarantee a low test error rate and a low rejection rate with respect to $P$. Our algorithm is efficient given an Empirical Risk Minimizer (ERM) for $C$. Our guarantees hold even for test examples chosen by an unbounded white-box adversary. We also give guarantees for generalization, agnostic, and unsupervised settings.