Conservative classifiers do consistently well with improving agents: characterizing statistical and online learning

TL;DR

This paper characterizes the learnability of classifiers when agents improve their behavior over time, providing new theoretical insights into proper and improper learning under various assumptions and settings.

Contribution

It introduces an asymmetric concept class framework and offers exact characterizations of learnability with agent improvements in multiple settings.

Findings

Proper learning with improvements characterized in realizable setting.

Improper learning achieved under mild generative assumptions.

Lower generalization error and mistake bounds in online and noisy settings.

Abstract

Machine learning is now ubiquitous in societal decision-making, for example in evaluating job candidates or loan applications, and it is increasingly important to take into account how classified agents will react to the learning algorithms. The majority of recent literature on strategic classification has focused on reducing and countering deceptive behaviors by the classified agents, but recent work of Attias et al. identifies surprising properties of learnability when the agents genuinely improve in order to attain the desirable classification, such as smaller generalization error than standard PAC-learning. In this paper we characterize so-called learnability with improvements across multiple new axes. We introduce an asymmetric variant of minimally consistent concept classes and use it to provide an exact characterization of proper learning with improvements in the realizable setting. While prior work studies learnability only under general, arbitrary agent improvement regions, we give positive results for more natural Euclidean ball improvement sets. In particular, we characterize improper learning under a mild generative assumption on the data distribution. We further show how to learn in more challenging settings, achieving lower generalization error under well-studied bounded noise models and obtaining mistake bounds in realizable and agnostic online learning. We resolve open questions posed by Attias et al. for both proper and improper learning.