Rademacher learning rates for iterated random functions

TL;DR

This paper develops data-dependent learning rate bounds for models trained on data generated by iterated random functions, extending learning theory to non-i.i.d. time-dependent processes.

Contribution

It introduces a uniform convergence result and learning rate bounds for models trained on iterated random functions, considering non-irreducible, non-aperiodic Markov chains.

Findings

Data-dependent learning rates based on Rademacher complexities.

Uniform convergence for sample error in iterated random functions.

Learnability of ERM with derived rate bounds.

Abstract

Most existing literature on supervised machine learning assumes that the training dataset is drawn from an i.i.d. sample. However, many real-world problems exhibit temporal dependence and strong correlations between the marginal distributions of the data-generating process, suggesting that the i.i.d. assumption is often unrealistic. In such cases, models naturally include time-series processes with mixing properties, as well as irreducible and aperiodic ergodic Markov chains. Moreover, the learning rates typically obtained in these settings are independent of the data distribution, which can lead to restrictive choices of hypothesis classes and suboptimal sample complexities for the learning algorithm. In this article, we consider the case where the training dataset is generated by an iterated random function (i.e., an iteratively defined time-homogeneous Markov chain) that is not necessarily irreducible or aperiodic. Under the assumption that the governing function is contractive with respect to its first argument and subject to certain regularity conditions on the hypothesis class, we first establish a uniform convergence result for the corresponding sample error. We then demonstrate the learnability of the approximate empirical risk minimization algorithm and derive its learning rate bound. Both rates are data-distribution dependent, expressed in terms of the Rademacher complexities of the underlying hypothesis class, allowing them to more accurately reflect the properties of the data-generating distribution.