Surrogate to Poincar\'e inequalities on manifolds for structured dimension reduction in nonlinear feature spaces

TL;DR

This paper develops a gradient-based method leveraging Poincaré inequalities on manifolds to construct structured nonlinear feature maps for high-dimensional function approximation, addressing challenges with small training samples.

Contribution

It introduces a quadratic surrogate for the non-convex loss and analyzes structured feature map construction in both collective and grouped settings.

Findings

Introduces a new quadratic surrogate for the non-convex loss functional.

Provides upper bounds on the surrogate loss in the collective setting.

Shows the equivalence between grouped and multiple collective feature map constructions.

Abstract

This paper is concerned with the approximation of continuously differentiable functions with high-dimensional input by a composition of two functions: a feature map that extracts few features from the input space, and a profile function that approximates the target function taking the features as its low-dimensional input. We focus on the construction of structured nonlinear feature maps, that extract features on separate groups of variables, using a recently introduced gradient-based method that leverages Poincar\'e inequalities on nonlinear manifolds. This method consists in minimizing a non-convex loss functional, which can be a challenging task, especially for small training samples. We first investigate a collective setting, in which we construct a feature map suitable to a parametrized family of high-dimensional functions. In this setting we introduce a new quadratic surrogate to the non-convex loss function and show an upper bound on the latter. We then investigate a grouped setting, in which we construct separate feature maps for separate groups of inputs, and we show that this setting is almost equivalent to multiple collective settings, one for each group of variables.