Probabilistic partition of unity networks for high-dimensional regression problems

TL;DR

This paper introduces PPOU-Nets, a probabilistic model for high-dimensional regression that combines mixture of experts with adaptive dimensionality reduction, outperforming standard neural networks in various data settings.

Contribution

The paper proposes a novel PPOU-Net framework that integrates EM-based training with local polynomial experts for improved high-dimensional regression.

Findings

PPOU-Nets outperform baseline neural networks in numerical experiments.

The model effectively reduces dimensionality in high-dimensional data.

Application to quantum computing demonstrates versatility as surrogate models.

Abstract

We explore the probabilistic partition of unity network (PPOU-Net) model in the context of high-dimensional regression problems and propose a general framework focusing on adaptive dimensionality reduction. With the proposed framework, the target function is approximated by a mixture of experts model on a low-dimensional manifold, where each cluster is associated with a local fixed-degree polynomial. We present a training strategy that leverages the expectation maximization (EM) algorithm. During the training, we alternate between (i) applying gradient descent to update the DNN coefficients; and (ii) using closed-form formulae derived from the EM algorithm to update the mixture of experts model parameters. Under the probabilistic formulation, step (ii) admits the form of embarrassingly parallelizable weighted least-squares solves. The PPOU-Nets consistently outperform the baseline fully-connected neural networks of comparable sizes in numerical experiments of various data dimensions. We also explore the proposed model in applications of quantum computing, where the PPOU-Nets act as surrogate models for cost landscapes associated with variational quantum circuits.