High-dimensional Adaptive MCMC with Reduced Computational Complexity

TL;DR

This paper introduces an adaptive MCMC method that learns a sparse dense preconditioner using eigeninformation, reducing computational complexity while improving performance on correlated target distributions.

Contribution

The authors develop a novel adaptive preconditioning strategy that balances complexity and correlation capture, outperforming existing methods in efficiency and effectiveness.

Findings

Outperforms diagonal preconditioning in correlated targets.

Reduces per-iteration complexity to O(m^2d) from O(d^2).

Achieves better time-normalized performance than dense preconditioning.

Abstract

We propose an adaptive MCMC method that learns a linear preconditioner which is dense in its off-diagonal elements but sparse in its parametrisation. Due to this sparsity, we achieve a per-iteration computational complexity of $O(m^2d)$ for a user-determined parameter $m$, compared with the $O(d^2)$ complexity of existing adaptive strategies that can capture correlation information from the target. Diagonal preconditioning has an $O(d)$ per-iteration complexity, but is known to fail in the case that the target distribution is highly correlated, see \citet[Section 3.5]{hird2025a}. Our preconditioner is constructed using eigeninformation from the target covariance which we infer using online principal components analysis on the MCMC chain. It is composed of a diagonal matrix and a product of carefully chosen reflection matrices. On various numerical tests we show that it outperforms diagonal preconditioning in terms of absolute performance, and that it outperforms traditional dense preconditioning and multiple diagonal plus low-rank alternatives in terms of time-normalised performance.