Adaptive Approximation and Generalization of Deep Neural Network with Intrinsic Dimensionality

TL;DR

This paper demonstrates that the performance of deep neural networks is primarily influenced by the intrinsic low dimensionality of data covariates, rather than their high ambient dimension, through theoretical bounds and simulations.

Contribution

The paper introduces a novel theoretical framework using Minkowski dimension to analyze DNNs' approximation and generalization errors based on intrinsic data dimension, showing optimal convergence rates.

Findings

Error bounds depend on intrinsic dimension, not ambient dimension

DNNs achieve optimal convergence rates in the minimax sense

Numerical simulations validate theoretical predictions

Abstract

In this study, we prove that an intrinsic low dimensionality of covariates is the main factor that determines the performance of deep neural networks (DNNs). DNNs generally provide outstanding empirical performance. Hence, numerous studies have actively investigated the theoretical properties of DNNs to understand their underlying mechanisms. In particular, the behavior of DNNs in terms of high-dimensional data is one of the most critical questions. However, this issue has not been sufficiently investigated from the aspect of covariates, although high-dimensional data have practically low intrinsic dimensionality. In this study, we derive bounds for an approximation error and a generalization error regarding DNNs with intrinsically low dimensional covariates. We apply the notion of the Minkowski dimension and develop a novel proof technique. Consequently, we show that convergence rates of the errors by DNNs do not depend on the nominal high dimensionality of data, but on its lower intrinsic dimension. We further prove that the rate is optimal in the minimax sense. We identify an advantage of DNNs by showing that DNNs can handle a broader class of intrinsic low dimensional data than other adaptive estimators. Finally, we conduct a numerical simulation to validate the theoretical results.