Robust deep learning from weakly dependent data

TL;DR

This paper develops non-asymptotic bounds for deep neural network estimators trained on weakly dependent, heavy-tailed data, demonstrating near-i.i.d. performance in certain smoothness conditions and validating robustness through simulations.

Contribution

It extends theoretical analysis of deep learning to unbounded, dependent data with heavy tails, providing convergence rates and robustness guarantees.

Findings

Non-asymptotic bounds established under weak dependence.

Robust estimators outperform least squares in heavy-tailed scenarios.

Convergence rates close to i.i.d. cases for smooth functions.

Abstract

Recent developments on deep learning established some theoretical properties of deep neural networks estimators. However, most of the existing works on this topic are restricted to bounded loss functions or (sub)-Gaussian or bounded input. This paper considers robust deep learning from weakly dependent observations, with unbounded loss function and unbounded input/output. It is only assumed that the output variable has a finite $r$ order moment, with $r >1$. Non asymptotic bounds for the expected excess risk of the deep neural network estimator are established under strong mixing, and $\psi$-weak dependence assumptions on the observations. We derive a relationship between these bounds and $r$, and when the data have moments of any order (that is $r=\infty$), the convergence rate is close to some well-known results. When the target predictor belongs to the class of H\"older smooth functions with sufficiently large smoothness index, the rate of the expected excess risk for exponentially strongly mixing data is close to or as same as those for obtained with i.i.d. samples. Application to robust nonparametric regression and robust nonparametric autoregression are considered. The simulation study for models with heavy-tailed errors shows that, robust estimators with absolute loss and Huber loss function outperform the least squares method.