Deep learning from strongly mixing observations: Sparse-penalized regularization and minimax optimality

TL;DR

This paper investigates deep learning with dependent, strongly mixing data, establishing regularization techniques and minimax optimality results for various loss functions and nonparametric regression scenarios.

Contribution

It extends the theory of deep neural network estimators to dependent data, providing oracle inequalities and minimax optimality results under strong mixing conditions.

Findings

Oracle inequality for expected excess risk.

Upper bound on L2 error for nonparametric regression.

Deep neural networks achieve minimax optimal rates.

Abstract

The explicit regularization and optimality of deep neural networks estimators from independent data have made considerable progress recently. The study of such properties on dependent data is still a challenge. In this paper, we carry out deep learning from strongly mixing observations, and deal with the squared and a broad class of loss functions. We consider sparse-penalized regularization for deep neural network predictor. For a general framework that includes, regression estimation, classification, time series prediction,$\cdots$, oracle inequality for the expected excess risk is established and a bound on the class of H\"older smooth functions is provided. For nonparametric regression from strong mixing data and sub-exponentially error, we provide an oracle inequality for the $L_2$ error and investigate an upper bound of this error on a class of H\"older composition functions. For the specific case of nonparametric autoregression with Gaussian and Laplace errors, a lower bound of the $L_2$ error on this H\"older composition class is established. Up to logarithmic factor, this bound matches its upper bound; so, the deep neural network estimator attains the minimax optimal rate.