High-Performance Statistical Computing (HPSC): Challenges, Opportunities, and Future Directions

TL;DR

This paper discusses the emergence of high-performance statistical computing (HPSC), highlighting challenges, opportunities, and a roadmap for integrating statistical methods with high-performance computing platforms to advance scientific research.

Contribution

It provides a historical overview, identifies key challenges, and proposes a vision and roadmap for developing a dedicated HPSC community bridging statistics and HPC.

Findings

HPSC is an emerging field with significant potential.

Bridging SC and HPC communities can accelerate statistical applications.

A roadmap for developing HPSC community is proposed.

Abstract

We recognize the emergence of a statistical computing community focused on working with large computing platforms and producing software and applications that exemplify high-performance statistical computing (HPSC). The statistical computing (SC) community develops software that is widely used across disciplines. However, it remains largely absent from the high-performance computing (HPC) landscape, particularly on platforms such as those featured on the Top500 or Green500 lists. Many disciplines already participate in HPC, mostly centered around simulation science, although data-focused efforts under the artificial intelligence (AI) label are gaining popularity. Bridging this gap requires both community adaptation and technical innovation to align statistical methods with modern HPC technologies. We can accelerate progress in fast and scalable statistical applications by building strong connections between the SC and HPC communities. We present a brief history of SC, a vision for how its strengths can contribute to statistical science in the HPC environment (such as HPSC), the challenges that remain, and the opportunities currently available, culminating in a possible roadmap toward a thriving HPSC community.