Data Compression with Noise Suppression for Inference under Noisy Covariance

TL;DR

This paper introduces p-MOPED, a novel data compression method that reduces noise in covariance estimates, improving parameter inference accuracy in noisy, high-dimensional data scenarios like cosmology.

Contribution

We propose p-MOPED, a modified compression scheme that suppresses noise propagation in covariance matrices, enhancing data analysis robustness under limited simulation conditions.

Findings

p-MOPED outperforms standard MOPED in noisy regimes

Effective noise suppression in covariance estimates

Improved parameter inference accuracy

Abstract

In many fields including cosmology, statistical inference often relies on Gaussian likelihoods whose covariance matrices are estimated from a finite number of simulations. This finite-sample estimation introduces noise into the covariance, which propagates to parameter estimates, a phenomenon known as the Dodelson-Schneider (DS) effect, leading to inflated uncertainties. While the Massively Optimized Parameter Estimation and Data compression (MOPED) algorithm offers lossless Fisher information-preserving compression, it does not mitigate the DS effect when the compression matrix itself is derived from noisy covariances. In this paper, we propose a modified compression scheme, powered MOPED ($p$-MOPED), which suppresses noise propagation by balancing information retention and covariance estimate noise reduction through a tunable power-law transformation of the sample correlation matrix. We test $p$-MOPED against standard and diagonal MOPED on toy models and on cosmological data from the Subaru Hyper Suprime-Cam Year 3 weak lensing survey. Our results demonstrate that $p$-MOPED consistently outperforms other approaches, especially in regimes with limited simulations, offering a robust compression strategy for high-dimensional data analyses under practical constraints.