Cosmological neutrino simulations at extreme scale

TL;DR

This paper presents the development and execution of the TianNu simulation, the world's largest cosmological N-body simulation incorporating neutrinos, leveraging extreme-scale high performance computing to improve neutrino clustering predictions.

Contribution

It introduces novel code optimizations and data compression techniques for large-scale neutrino simulations, enabling unprecedented simulation scale and precision.

Findings

TianNu simulation uses 2.97 trillion particles, the largest of its kind.

Achieved significant reduction in data footprint from 24 bytes to 9 bytes per particle.

Successfully scaled the simulation to the Tianhe-2 supercomputer, utilizing 86% of its capacity.

Abstract

Constraining neutrino mass remains an elusive challenge in modern physics. Precision measurements are expected from several upcoming cosmological probes of large-scale structure. Achieving this goal relies on an equal level of precision from theoretical predictions of neutrino clustering. Numerical simulations of the non-linear evolution of cold dark matter and neutrinos play a pivotal role in this process. We incorporate neutrinos into the cosmological N-body code CUBEP3M and discuss the challenges associated with pushing to the extreme scales demanded by the neutrino problem. We highlight code optimizations made to exploit modern high performance computing architectures and present a novel method of data compression that reduces the phase-space particle footprint from 24 bytes in single precision to roughly 9 bytes. We scale the neutrino problem to the Tianhe-2 supercomputer and provide details of our production run, named TianNu, which uses 86% of the machine (13,824 compute nodes). With a total of 2.97 trillion particles, TianNu is currently the world's largest cosmological N-body simulation and improves upon previous neutrino simulations by two orders of magnitude in scale. We finish with a discussion of the unanticipated computational challenges that were encountered during the TianNu runtime.