On the Consistency of Kernel Methods with Dependent Observations

TL;DR

This paper introduces empirical weak convergence (EWC) as a new, less restrictive assumption to establish the consistency of kernel methods like SVMs and Gaussian processes on dependent data, broadening their theoretical foundation.

Contribution

It proposes EWC as a novel assumption that explains kernel method performance on dependent data and proves their consistency under this weaker condition, extending classical results.

Findings

Establishes consistency of SVMs and kernel mean embeddings under EWC

Extends classical statistical learning results to infinite-dimensional outputs

Provides a foundation for learning beyond i.i.d. and mixing assumptions

Abstract

The consistency of a learning method is usually established under the assumption that the observations are a realization of an independent and identically distributed (i.i.d.) or mixing process. Yet, kernel methods such as support vector machines (SVMs), Gaussian processes, or conditional kernel mean embeddings (CKMEs) all give excellent performance under sampling schemes that are obviously non-i.i.d., such as when data comes from a dynamical system. We propose the new notion of empirical weak convergence (EWC) as a general assumption explaining such phenomena for kernel methods. It assumes the existence of a random asymptotic data distribution and is a strict weakening of previous assumptions in the field. Our main results then establish consistency of SVMs, kernel mean embeddings, and general Hilbert-space valued empirical expectations with EWC data. Our analysis holds for both finite- and infinite-dimensional outputs, as we extend classical results of statistical learning to the latter case. In particular, it is also applicable to CKMEs. Overall, our results open new classes of processes to statistical learning and can serve as a foundation for a theory of learning beyond i.i.d. and mixing.