libEnsemble: A Library to Coordinate the Concurrent Evaluation of Dynamic Ensembles of Calculations

TL;DR

libEnsemble is a Python library designed to coordinate concurrent evaluations of dynamic ensembles, addressing scalability challenges and enhancing performance for large-scale computational applications in exascale computing environments.

Contribution

The paper introduces libEnsemble, a novel library that enables scalable, concurrent execution of dynamic ensembles of calculations, improving performance on pre-exascale and exascale systems.

Findings

Successfully demonstrated on pre-exascale environments

Supports advanced capabilities for exascale computing

Enhances scalability and user functionality in high-performance computing

Abstract

Almost all applications stop scaling at some point; those that don't are seldom performant when considering time to solution on anything but aspirational/unicorn resources. Recognizing these tradeoffs as well as greater user functionality in a near-term exascale computing era, we present libEnsemble, a library aimed at particular scalability- and capability-stretching uses. libEnsemble enables running concurrent instances of an application in dynamically allocated ensembles through an extensible Python library. We highlight the structure, execution, and capabilities of the library on leading pre-exascale environments as well as advanced capabilities for exascale environments and beyond.