Estimating composite functions by model selection

TL;DR

This paper introduces a model selection approach for estimating high-dimensional functions by approximating them with composite functions, enabling adaptive estimation across various models and smoothness classes.

Contribution

It develops a general, flexible method for estimating composite functions that adapts to different models and regularity conditions, including neural networks and additive models.

Findings

Applicable to various models like neural networks and additive functions

Provides adaptive estimators for functions with different smoothness levels

Achieves broad approximation capabilities in high-dimensional settings

Abstract

We consider the problem of estimating a function $s$ on $[-1,1]^{k}$ for large values of $k$ by looking for some best approximation by composite functions of the form $g\circ u$. Our solution is based on model selection and leads to a very general approach to solve this problem with respect to many different types of functions $g,u$ and statistical frameworks. In particular, we handle the problems of approximating $s$ by additive functions, single and multiple index models, neural networks, mixtures of Gaussian densities (when $s$ is a density) among other examples. We also investigate the situation where $s=g\circ u$ for functions $g$ and $u$ belonging to possibly anisotropic smoothness classes. In this case, our approach leads to a completely adaptive estimator with respect to the regularity of $s$.