Scaling pair count to next galaxy surveys

TL;DR

This paper introduces an efficient, scalable algorithm using Apache Spark and a novel sphere pixelization method to rapidly compute galaxy pair counts in massive datasets, addressing big data challenges in upcoming cosmological surveys.

Contribution

The paper presents a new sphere pixelization scheme (SARSPix) and an optimized Spark-based algorithm for fast pair counting in billion-scale galaxy datasets, improving computational efficiency.

Findings

Achieves pair-distance histogram computation in about 2 minutes for billion-scale data

Demonstrates scalability over 16 to 64 nodes in a Spark cluster

Provides publicly available software for large-scale galaxy data analysis

Abstract

Counting pairs of galaxies or stars according to their distance is at the core of real-space correlation analyzes performed in astrophysics and cosmology. Upcoming galaxy surveys (LSST, Euclid) will measure properties of billions of galaxies challenging our ability to perform such counting in a minute-scale time relevant for the usage of simulations. The problem is only limited by efficient access to the data, hence belongs to the big data category. We use the popular Apache Spark framework to address it and design an efficient high-throughput algorithm to deal with hundreds of millions to billions of input data. To optimize it, we revisit the question of nonhierarchical sphere pixelization based on cube symmetries and develop a new one dubbed the "Similar Radius Sphere Pixelization" (SARSPix) with very close to square pixels. It provides the most adapted indexing over the sphere for all distance-related computations. Using LSST-like fast simulations, we compute autocorrelation functions on tomographic bins containing between a hundred million to one billion data points. In each case we achieve the construction of a standard pair-distance histogram in about 2 minutes, using a simple algorithm that is shown to scale, over a moderate number of nodes (16 to 64). This illustrates the potential of this new techniques in the field of astronomy where data access is becoming the main bottleneck. They can be easily adapted to other use-cases as nearest-neighbors search, catalog cross-match or cluster finding. The software is publicly available from https://github.com/astrolabsoftware/SparkCorr.