CosmoHammer: Cosmological parameter estimation with the MCMC Hammer

TL;DR

This paper introduces CosmoHammer, a parallelised MCMC framework that significantly accelerates cosmological parameter estimation, enabling rapid analysis of complex data sets using cloud computing resources.

Contribution

The paper presents CosmoHammer, a Python framework that efficiently distributes MCMC sampling across thousands of cores, reducing computation time from hours to minutes for cosmological data analysis.

Findings

CosmoHammer reduces sampling time from 30 hours to 16 minutes using 2048 cores.

Parallelisation significantly improves efficiency of MCMC in cosmology.

Framework enables extensive model testing and systematic control.

Abstract

We study the benefits and limits of parallelised Markov chain Monte Carlo (MCMC) sampling in cosmology. MCMC methods are widely used for the estimation of cosmological parameters from a given set of observations and are typically based on the Metropolis-Hastings algorithm. Some of the required calculations can however be computationally intensive, meaning that a single long chain can take several hours or days to calculate. In practice, this can be limiting, since the MCMC process needs to be performed many times to test the impact of possible systematics and to understand the robustness of the measurements being made. To achieve greater speed through parallelisation, MCMC algorithms need to have short auto-correlation times and minimal overheads caused by tuning and burn-in. The resulting scalability is hence influenced by two factors, the MCMC overheads and the parallelisation costs. In order to efficiently distribute the MCMC sampling over thousands of cores on modern cloud computing infrastructure, we developed a Python framework called CosmoHammer which embeds emcee, an implementation by Foreman-Mackey et al. (2012) of the affine invariant ensemble sampler by Goodman and Weare (2010). We test the performance of CosmoHammer for cosmological parameter estimation from cosmic microwave background data. While Metropolis-Hastings is dominated by overheads, CosmoHammer is able to accelerate the sampling process from a wall time of 30 hours on a dual core notebook to 16 minutes by scaling out to 2048 cores. Such short wall times for complex data sets opens possibilities for extensive model testing and control of systematics.