BeeSwarm: Enabling Scalability Tests in Continuous Integration

TL;DR

BeeSwarm is a new system that integrates scalability testing into existing continuous integration environments, facilitating HPC developers to easily perform performance and scalability assessments within their CI workflows.

Contribution

We introduce BeeSwarm, a scalable testing system that seamlessly integrates with popular CI tools, enabling automated scalability testing for HPC applications.

Findings

Successfully integrated into Travis CI and GitLab CI.

Enabled scalable testing workflows on Chameleon cloud.

Improved accessibility of scalability testing in CI environments.

Abstract

Testing is one of the most important steps in software development. It ensures the quality of software. Continuous Integration (CI) is a widely used testing system that can report software quality to the developer in a timely manner during the development progress. Performance, especially scalability, is another key factor for High Performance Computing (HPC) applications. Though there are many applications and tools to profile the performance of HPC applications, none of them are integrated into the continuous integration. On the other hand, no current continuous integration tools provide easy-to-use scalability test capabilities. In this work, we propose BeeSwarm, a scalability test system that can be easily applied to the current CI test environment enabling scalability test capability for HPC developers. As a showcase, BeeSwarm is integrated into Travis CI and GitLab CI to execute the scalability test workflow on Chameleon cloud.