Deploying a sharded MongoDB cluster as a queued job on a shared HPC architecture

TL;DR

This paper demonstrates deploying a sharded MongoDB cluster as a queued job on an HPC system, enabling concurrent data science workloads and measuring performance across configurations on Blue Waters supercomputer.

Contribution

It introduces a method to run a MongoDB sharded cluster within an HPC job queue, integrating data store deployment with HPC scheduling for data science tasks.

Findings

Successful deployment of MongoDB cluster on HPC system

Performance measurements for data ingest and queries

Insights into configuration impacts on HPC performance

Abstract

Data stores are the foundation on which data science, in all its variations, is built upon. They provide a queryable interface to structured and unstructured data. Data science often starts by leveraging these query features to perform initial data preparation. However, most data stores are designed to run continuously to service disparate user requests with little or no downtime. Many HPC architectures process user requests by job queue scheduler and maintain a shard filesystem to store a jobs persistent data. We deploy a MongoDB sharded cluster with a run script that is designed to run a data science workload concurrently. As our test piece, we run data ingest and data queries to measure the performance with different configurations on the Blue Waters supper computer.