Learning from dependent observations

TL;DR

This paper extends the theoretical understanding of support vector machines (SVMs) by demonstrating their effectiveness under dependent data processes, such as mixing processes, beyond the traditional i.i.d. assumption.

Contribution

It shows that SVMs require only a law of large numbers for data, enabling learnability results for dependent and non-stationary processes, including unbounded noise in regression.

Findings

SVMs are effective under certain dependent data conditions.

Learnability results hold for non-i.i.d. and non-stationary processes.

Applicable to classification and regression with unbounded noise.

Abstract

In most papers establishing consistency for learning algorithms it is assumed that the observations used for training are realizations of an i.i.d. process. In this paper we go far beyond this classical framework by showing that support vector machines (SVMs) essentially only require that the data-generating process satisfies a certain law of large numbers. We then consider the learnability of SVMs for $\a$-mixing (not necessarily stationary) processes for both classification and regression, where for the latter we explicitly allow unbounded noise.