Universally Consistent Online Learning with Arbitrarily Dependent Responses

TL;DR

This paper introduces an online learning rule that guarantees universal consistency for a broad class of processes, including non-ergodic and non-stationary ones, focusing only on the properties of the X process.

Contribution

It generalizes previous results by removing the need for stationarity and ergodicity of the joint process, relying solely on conditions on the X process.

Findings

Universal consistency achieved under minimal process conditions

Ergodicity is shown to be unnecessary for online learning consistency

Applicable to non-stationary, dependent data processes

Abstract

This work provides an online learning rule that is universally consistent under processes on (X,Y) pairs, under conditions only on the X process. As a special case, the conditions admit all processes on (X,Y) such that the process on X is stationary. This generalizes past results which required stationarity for the joint process on (X,Y), and additionally required this process to be ergodic. In particular, this means that ergodicity is superfluous for the purpose of universally consistent online learning.