Deep Neural Networks for Nonparametric Interaction Models with Diverging Dimension

TL;DR

This paper analyzes deep neural networks for nonparametric interaction models in high and diverging dimensions, introducing a debiasing technique to achieve optimal convergence rates and handle covariance issues.

Contribution

It introduces a novel debiasing method for deep neural networks to effectively manage covariance terms in high-dimensional nonparametric models, achieving minimax optimal rates.

Findings

Debiased neural networks attain minimax optimal convergence rates.

Covariance among additive components can be controlled with the proposed debiasing technique.

The method is effective in both growing and high-dimensional regimes.

Abstract

Deep neural networks have achieved tremendous success due to their representation power and adaptation to low-dimensional structures. Their potential for estimating structured regression functions has been recently established in the literature. However, most of the studies require the input dimension to be fixed and consequently ignore the effect of dimension on the rate of convergence and hamper their applications to modern big data with high dimensionality. In this paper, we bridge this gap by analyzing a $k^{th}$ order nonparametric interaction model in both growing dimension scenarios ($d$ grows with $n$ but at a slower rate) and in high dimension ($d \gtrsim n$). In the latter case, sparsity assumptions and associated regularization are required in order to obtain optimal rates of convergence. A new challenge in diverging dimension setting is in calculation mean-square error, the covariance terms among estimated additive components are an order of magnitude larger than those of the variances and they can deteriorate statistical properties without proper care. We introduce a critical debiasing technique to amend the problem. We show that under certain standard assumptions, debiased deep neural networks achieve a minimax optimal rate both in terms of $(n, d)$. Our proof techniques rely crucially on a novel debiasing technique that makes the covariances of additive components negligible in the mean-square error calculation. In addition, we establish the matching lower bounds.