Continuous benchmarking: Keeping pace with an evolving ecosystem of models and technologies

TL;DR

This paper introduces an automated, continuous benchmarking pipeline designed to adapt to rapidly evolving models and technologies, emphasizing reproducibility, collaboration, and sustainability in high-performance computing for neuroscience and AI.

Contribution

It extends previous benchmarking workflows with user-agnostic and continuous benchmarking features, facilitating sustainable progress in research software development.

Findings

Enhanced reproducibility and re-use of benchmarking results

Software solutions for rapid adaptation to evolving models and systems

Support for collaborative and customizable benchmarking workflows

Abstract

Drawing on ideas from continuous integration, we present concepts of an automated benchmarking pipeline for high performance applications. Customization and collaboration have been key design goals owing to the requirements of research-software development as a continuous community effort. We have extended our previous conceptual work on systematic benchmarking workflows with the functionality of user-agnostic operations as well as continuous benchmarking. This fosters reproducibility and re-use of benchmarking results to ensure sustainable technological progress. We provide software-engineering solutions to keep pace with the rapid evolution of both large-scale models and high-performance computing systems with a view towards the scientific domains of neuroscience and artificial intelligence.