Scalable Streaming Tools for Analyzing $N$-body Simulations: Finding Halos and Investigating Excursion Sets in One Pass

TL;DR

This paper introduces a scalable GPU-accelerated streaming tool for analyzing extremely large cosmological N-body simulation datasets, enabling efficient halo detection and statistical analysis on datasets with up to 10^12 particles.

Contribution

It presents a novel, high-performance streaming algorithm leveraging GPU, sampling, and parallel I/O to analyze large-scale N-body simulations beyond previous limits.

Findings

Scales to datasets with up to 10^12 particles

Detects 10^4-10^5 halo centers efficiently

Operates within an hour on a single GPU

Abstract

Cosmological $N$-body simulations play a vital role in studying models for the evolution of the Universe. To compare to observations and make a scientific inference, statistic analysis on large simulation datasets, e.g., finding halos, obtaining multi-point correlation functions, is crucial. However, traditional in-memory methods for these tasks do not scale to the datasets that are forbiddingly large in modern simulations. Our prior paper proposes memory-efficient streaming algorithms that can find the largest halos in a simulation with up to $10^9$ particles on a small server or desktop. However, this approach fails when directly scaling to larger datasets. This paper presents a robust streaming tool that leverages state-of-the-art techniques on GPU boosting, sampling, and parallel I/O, to significantly improve performance and scalability. Our rigorous analysis of the sketch parameters improves the previous results from finding the centers of the $10^3$ largest halos to $\sim 10^4-10^5$, and reveals the trade-offs between memory, running time and number of halos. Our experiments show that our tool can scale to datasets with up to $\sim 10^{12}$ particles while using less than an hour of running time on a single GPU Nvidia GTX 1080.