Analysis of the rate of convergence of neural network regression estimates which are easy to implement

TL;DR

This paper introduces an easy-to-implement neural network regression method that achieves near-optimal convergence rates for smooth functions, combining neural networks with random projection pursuit techniques.

Contribution

It proposes a new neural network regression estimate with fixed weights based on recent approximation results, simplifying implementation while maintaining theoretical convergence guarantees.

Findings

Achieves near-optimal convergence rates for smooth regression functions.

Combines neural networks with random projection pursuit for practical implementation.

Demonstrates the method's effectiveness through theoretical convergence results.

Abstract

Recent results in nonparametric regression show that for deep learning, i.e., for neural network estimates with many hidden layers, we are able to achieve good rates of convergence even in case of high-dimensional predictor variables, provided suitable assumptions on the structure of the regression function are imposed. The estimates are defined by minimizing the empirical $L_2$ risk over a class of neural networks. In practice it is not clear how this can be done exactly. In this article we introduce a new neural network regression estimate where most of the weights are chosen regardless of the data motivated by some recent approximation results for neural networks, and which is therefore easy to implement. We show that for this estimate we can derive rates of convergence results in case the regression function is smooth. We combine this estimate with the projection pursuit, where we choose the directions randomly, and we show that for sufficiently many repititions we get a neural network regression estimate which is easy to implement and which achieves the one-dimensional rate of convergence (up to some logarithmic factor) in case that the regression function satisfies the assumptions of projection pursuit.