Concentration Inequalities for Suprema of Empirical Processes with Dependent Data via Generic Chaining with Applications to Statistical Learning

TL;DR

This paper introduces a new concentration inequality for the supremum of empirical processes with dependent data, leveraging generic chaining and coupling techniques, applicable to high-dimensional, heavy-tailed data, and providing guarantees for empirical risk minimization.

Contribution

It develops a novel concentration inequality framework for dependent data using generic chaining and coupling, extending analysis to high-dimensional and heavy-tailed scenarios.

Findings

Provides non-asymptotic guarantees for empirical risk minimization with dependent data.

Establishes an oracle inequality for nonlinear regression models.

Shows prediction accuracy comparable to i.i.d. data in dependent settings.

Abstract

This paper develops a general concentration inequality for the suprema of empirical processes with dependent data. The concentration inequality is obtained by combining generic chaining with a coupling-based strategy. Our framework accommodates high-dimensional and heavy-tailed (sub-Weibull) data. We demonstrate the usefulness of our result by deriving non-asymptotic predictive performance guarantees for empirical risk minimization in regression problems with dependent data. In particular, we establish an oracle inequality for a broad class of nonlinear regression models and, as a special case, a single-layer neural network model. Our results show that empirical risk minimzaton with dependent data attains a prediction accuracy comparable to that in the i.i.d. setting for a wide range of nonlinear regression models.