Confronting missing observations with probability weights: Fourier space and generalised formalism

TL;DR

This paper develops a comprehensive statistical formalism for using probability weights to correct for missing observations in galaxy redshift surveys, improving the estimation of clustering statistics and cosmological parameters.

Contribution

It introduces a mature formalism for probability weights, enabling efficient inverse probability estimation and extending traditional missing data corrections in cosmological surveys.

Findings

Derived an inverse-probability-based power spectrum estimator.

Enhanced robustness and efficiency of existing configuration-space estimators.

Demonstrated effectiveness using idealised survey simulations.

Abstract

Due to instrumental limitations, the nature of which vary from case to case, spectroscopic galaxy redshift surveys usually do not collect redshifts for all galaxies in the population of potential targets. Especially problematic is the entanglement between this incompleteness and the true cosmological signal, arising from the fact that the proportion of successful observations is typically lower in regions with higher density of galaxies. The result is a fictitious suppression of the galaxy clustering that, if not properly corrected, can impact severely on cosmological-parameter inference. Recent developments in the field have shown that an unbiased estimate of the 2-point correlation function in the presence of missing observations can be obtained by weighting each pair by its inverse probability of being targeted. In this work we expand on the concept of probability weights by developing a more mature statistical formalism, which provides us with a deeper understanding of their fundamental properties. We take advantage of this novel perspective to handle the problem of estimating the inverse probability, specifically, we discuss how to efficiently determine the weights from a finite set of realisations of the targeting and how to model exactly the resulting sampling effects. This allows us to derive an inverse-probability-based estimator for the power spectrum, which is the main result of this work, but also to improve robustness and computational efficiency of the already existing configuration-space estimator. Finally, we propose a strategy to further extend the concept of inverse probability, providing examples of how traditional missing-observation countermeasures can be included in this more general picture. The effectiveness of the different models and weighting schemes discussed in this work is demonstrated using realisations of an idealised simple survey strategy.