Scaling Big Data Platform for Big Data Pipeline

TL;DR

This paper presents a scalable 3D visualization platform for monitoring and managing supercomputer systems, leveraging Accumulo, d4m, and Unity to handle the growing data volume efficiently.

Contribution

It introduces a novel approach to scale visualization tools for high-performance computing systems using advanced database and visualization technologies.

Findings

Successfully scaled the visualization platform to handle large HPC data.

Enabled proactive system failure detection through real-time visualization.

Improved system management efficiency with the new platform.

Abstract

Monitoring and Managing High Performance Computing (HPC) systems and environments generate an ever growing amount of data. Making sense of this data and generating a platform where the data can be visualized for system administrators and management to proactively identify system failures or understand the state of the system requires the platform to be as efficient and scalable as the underlying database tools used to store and analyze the data. In this paper we will show how we leverage Accumulo, d4m, and Unity to generate a 3D visualization platform to monitor and manage the Lincoln Laboratory Supercomputer systems and how we have had to retool our approach to scale with our systems.