Dense ReLU Neural Networks for Temporal-spatial Model

TL;DR

This paper develops a theoretical framework and empirical validation for dense ReLU neural networks in modeling complex temporal-spatial data, addressing dependence, high dimensionality, and demonstrating superior performance.

Contribution

It introduces non-asymptotic bounds and convergence rates for neural networks handling temporal-spatial dependence, extending theoretical analysis to more general contexts and demonstrating empirical superiority.

Findings

Neural networks outperform existing methods in synthetic experiments.

Theoretical bounds are established for models with temporal-spatial dependence.

Modeling on manifolds mitigates the curse of dimensionality.

Abstract

In this paper, we focus on fully connected deep neural networks utilizing the Rectified Linear Unit (ReLU) activation function for nonparametric estimation. We derive non-asymptotic bounds that lead to convergence rates, addressing both temporal and spatial dependence in the observed measurements. By accounting for dependencies across time and space, our models better reflect the complexities of real-world data, enhancing both predictive performance and theoretical robustness. We also tackle the curse of dimensionality by modeling the data on a manifold, exploring the intrinsic dimensionality of high-dimensional data. We broaden existing theoretical findings of temporal-spatial analysis by applying them to neural networks in more general contexts and demonstrate that our proof techniques are effective for models with short-range dependence. Our empirical simulations across various synthetic response functions underscore the superior performance of our method, outperforming established approaches in the existing literature. These findings provide valuable insights into the strong capabilities of dense neural networks (Dense NN) for temporal-spatial modeling across a broad range of function classes.