Trans-dimensional Hamiltonian model selection and parameter estimation from sparse, noisy data

TL;DR

This paper introduces a Bayesian hybrid MCMC framework for estimating parameters and model dimension from sparse, noisy data, effectively addressing ill-posed inverse problems in high-dimensional, nonlinear settings.

Contribution

The authors develop a novel hybrid MCMC approach combining reversible-jump and parallel tempering techniques for joint parameter and model dimension estimation from limited data.

Findings

The method recovers meaningful posterior distributions with an order of magnitude less data than existing methods.

Application to quantum spin systems validates the approach on experimental data.

Framework is broadly applicable to ill-posed inverse problems in various scientific fields.

Abstract

High-throughput characterization often requires estimating parameters and model dimension from experimental data of limited quantity and quality. Such data may result in an ill-posed inverse problem, where multiple sets of parameters and model dimensions are consistent with available data. This ill-posed regime may render traditional machine learning and deterministic methods unreliable or intractable, particularly in high-dimensional, nonlinear, and mixed continuous and discrete parameter spaces. To address these challenges, we present a Bayesian framework that hybridizes several Markov chain Monte Carlo (MCMC) sampling techniques to estimate both parameters and model dimension from sparse, noisy data. By integrating sampling for mixed continuous and discrete parameter spaces, reversible-jump MCMC to estimate model dimension, and parallel tempering to accelerate exploration of complex posteriors, our approach enables principled parameter estimation and model selection in data-limited regimes. We apply our framework to a specific ill-posed problem in quantum information science: recovering the locations and hyperfine couplings of nuclear spins surrounding a spin-defect in a semiconductor from sparse, noisy coherence data. We show that a hybridized MCMC method can recover meaningful posterior distributions over physical parameters using an order of magnitude less data than existing approaches, and we validate our results on experimental measurements. More generally, our work provides a flexible, extensible strategy for solving a broad class of ill-posed inverse problems under realistic experimental constraints.