Universal Regression with Adversarial Responses

TL;DR

This paper develops algorithms for universal regression with adversarial responses in non-i.i.d. settings on general metric spaces, establishing conditions for learnability, and introducing optimistically universal learning rules.

Contribution

It introduces new algorithms and characterizations for universal regression in adversarial, non-i.i.d. environments on general metric spaces, with minimal assumptions and broad applicability.

Findings

Universal consistency achievable beyond stationary processes

Fundamental dichotomy in value spaces for mean estimation

Existence of optimistically universal learning rules

Abstract

We provide algorithms for regression with adversarial responses under large classes of non-i.i.d. instance sequences, on general separable metric spaces, with provably minimal assumptions. We also give characterizations of learnability in this regression context. We consider universal consistency which asks for strong consistency of a learner without restrictions on the value responses. Our analysis shows that such an objective is achievable for a significantly larger class of instance sequences than stationary processes, and unveils a fundamental dichotomy between value spaces: whether finite-horizon mean estimation is achievable or not. We further provide optimistically universal learning rules, i.e., such that if they fail to achieve universal consistency, any other algorithms will fail as well. For unbounded losses, we propose a mild integrability condition under which there exist algorithms for adversarial regression under large classes of non-i.i.d. instance sequences. In addition, our analysis also provides a learning rule for mean estimation in general metric spaces that is consistent under adversarial responses without any moment conditions on the sequence, a result of independent interest.