A Serverless Architecture for Efficient and Scalable Monte Carlo Markov Chain Computation

TL;DR

This paper presents a cloud-based serverless architecture that enables efficient, scalable, and low-overhead parallel computation of MCMC methods, demonstrated through an astronomical data analysis case.

Contribution

It introduces a novel serverless framework for parallel MCMC computation that reduces overhead and maintenance efforts compared to traditional infrastructure.

Findings

Significantly reduced computation time with thousands of processes.

Overhead grows logarithmically with the number of processes.

Eliminates need for infrastructure maintenance and code adaptation.

Abstract

Computer power is a constantly increasing demand in scientific data analyses, in particular when Markov Chain Monte Carlo (MCMC) methods are involved, for example for estimating integral functions or Bayesian posterior probabilities. In this paper, we describe the benefits of a parallel computation of MCMC using a cloud-based, serverless architecture: first, the computation time can be spread over thousands of processes, hence greatly reducing the time the user should wait to have its computation completed. Second, the overhead time required for running in parallel several processes is minor and grows logarithmically with respect to the number of processes. Third, the serverless approach does not require time-consuming efforts for maintaining and updating the computing infrastructure when/if the number of walkers increases or for adapting the code to optimally use the infrastructure. The benefits are illustrated with the computation of the posterior probability distribution of a real astronomical analysis.