Surrogate to Poincar\'e inequalities on manifolds for dimension reduction in nonlinear feature spaces

TL;DR

This paper introduces convex surrogate functions for Poincaré inequality-based loss to improve dimension reduction in nonlinear feature spaces, demonstrating better performance especially with small training data.

Contribution

It proposes new convex surrogates for the challenging Poincaré inequality loss, enabling more efficient optimization in manifold-based dimension reduction.

Findings

Outperforms standard methods in approximation errors.

Effective with small training sample sizes.

Applicable to a broad class of functions and measures.

Abstract

We aim to approximate a continuously differentiable function $u:\mathbb{R}^d \rightarrow \mathbb{R}$ by a composition of functions $f\circ g$ where $g:\mathbb{R}^d \rightarrow \mathbb{R}^m$, $m\leq d$, and $f : \mathbb{R}^m \rightarrow \mathbb{R}$ are built in a two stage procedure. For a fixed $g$, we build $f$ using classical regression methods, involving evaluations of $u$. Recent works proposed to build a nonlinear $g$ by minimizing a loss function $\mathcal{J}(g)$ derived from Poincar\'e inequalities on manifolds, involving evaluations of the gradient of $u$. A problem is that minimizing $\mathcal{J}$ may be a challenging task. Hence in this work, we introduce new convex surrogates to $\mathcal{J}$. Leveraging concentration inequalities, we provide suboptimality results for a class of functions $g$, including polynomials, and a wide class of input probability measures. We investigate performances on different benchmarks for various training sample sizes. We show that our approach outperforms standard iterative methods for minimizing the training Poincar\'e inequality based loss, often resulting in better approximation errors, especially for small training sets and $m=1$.