Statistical Properties of Deep Neural Networks with Dependent Data

TL;DR

This paper analyzes the statistical properties of deep neural networks when trained on dependent data, providing convergence rates and error bounds applicable to common DNN architectures in regression and classification tasks.

Contribution

It introduces general results for nonparametric sieve estimators that are directly applicable to DNNs under dependent data, including convergence rates and error bounds.

Findings

Established convergence rates for DNN estimators with nonstationary data.

Derived non-asymptotic error bounds for stationary $eta$-mixing data.

Applicable to common fully connected feedforward networks with growing width and depth.

Abstract

This paper establishes statistical properties of deep neural network (DNN) estimators under dependent data. Two general results for nonparametric sieve estimators directly applicable to DNN estimators are given. The first establishes rates for convergence in probability under nonstationary data. The second provides non-asymptotic probability bounds on $\mathcal{L}^{2}$-errors under stationary $\beta$-mixing data. I apply these results to DNN estimators in both regression and classification contexts imposing only a standard H\"older smoothness assumption. The DNN architectures considered are common in applications, featuring fully connected feedforward networks with any continuous piecewise linear activation function, unbounded weights, and a width and depth that grows with sample size. The framework provided also offers potential for research into other DNN architectures and time-series applications.