TaPS: A Performance Evaluation Suite for Task-based Execution Frameworks

TL;DR

TaPS is a comprehensive evaluation suite designed to standardize performance assessment of task-based execution frameworks across various applications and hardware, facilitating fair comparison and research advancement.

Contribution

It introduces a modular interface and reference applications for benchmarking task-based execution frameworks, addressing the lack of evaluation standards.

Findings

TaPS enables consistent performance evaluation across frameworks.

It supports synthetic and real-world application benchmarking.

Demonstrated through a survey of benchmark results.

Abstract

Task-based execution frameworks, such as parallel programming libraries, computational workflow systems, and function-as-a-service platforms, enable the composition of distinct tasks into a single, unified application designed to achieve a computational goal. Task-based execution frameworks abstract the parallel execution of an application's tasks on arbitrary hardware. Research into these task executors has accelerated as computational sciences increasingly need to take advantage of parallel compute and/or heterogeneous hardware. However, the lack of evaluation standards makes it challenging to compare and contrast novel systems against existing implementations. Here, we introduce TaPS, the Task Performance Suite, to support continued research in parallel task executor frameworks. TaPS provides (1) a unified, modular interface for writing and evaluating applications using arbitrary execution frameworks and data management systems and (2) an initial set of reference synthetic and real-world science applications. We discuss how the design of TaPS supports the reliable evaluation of frameworks and demonstrate TaPS through a survey of benchmarks using the provided reference applications.