Dimension reduction via score ratio matching

TL;DR

This paper introduces a score-matching based framework for dimension reduction in Bayesian inference that does not require gradients, enabling efficient analysis of high-dimensional problems with limited data.

Contribution

It extends gradient-based dimension reduction techniques to gradient-free settings using score ratio learning and eigenvalue deflation methods.

Findings

Outperforms standard score-matching in low-dimensional structured problems.

Effective for PDE-constrained Bayesian inverse problems.

Improves accuracy of low-dimensional basis identification with limited data.

Abstract

Gradient-based dimension reduction decreases the cost of Bayesian inference and probabilistic modeling by identifying maximally informative (and informed) low-dimensional projections of the data and parameters, allowing high-dimensional problems to be reformulated as cheaper low-dimensional problems. A broad family of such techniques identify these projections and provide error bounds on the resulting posterior approximations, via eigendecompositions of certain diagnostic matrices. Yet these matrices require gradients or even Hessians of the log-likelihood, excluding the purely data-driven setting and many problems of simulation-based inference. We propose a framework, derived from score-matching, to extend gradient-based dimension reduction to problems where gradients are unavailable. Specifically, we formulate an objective function to directly learn the score ratio function needed to compute the diagnostic matrices, propose a tailored parameterization for the score ratio network, and introduce regularization methods that capitalize on the hypothesized low-dimensional structure. We also introduce a novel algorithm to iteratively identify the low-dimensional reduced basis vectors more accurately with limited data based on eigenvalue deflation methods. We show that our approach outperforms standard score-matching for problems with low-dimensional structure, and demonstrate its effectiveness for PDE-constrained Bayesian inverse problems and conditional generative modeling.