Literature Study on Operational Data Analytics Frameworks in Large-scale Computing Infrastructures

TL;DR

This paper reviews existing operational data analytics frameworks for large-scale computing infrastructures, proposes a comprehensive new framework, and compares it to state-of-the-art solutions to highlight its novelty and benefits.

Contribution

It introduces a holistic ODA framework tailored for large-scale infrastructures, extending previous models and providing a comparative analysis with current frameworks.

Findings

Operational efficiencies achieved by state-of-the-art ODA frameworks

The proposed holistic framework covers multiple infrastructure layers

Comparison highlights the novelty and advantages of the new framework

Abstract

By 2025, there are zettabytes of data generated every year. The size and complexity of modern large-scale computing infrastructures like High-Performance Computing (HPC) systems continue to evolve and become complex, leaving us wondering about their manageability and sustainability concerns. Because of this reason, those complex systems are provided with fine-grained monitoring and Operational Data Analytics (ODA) capabilities to optimise their efficiency. In this literature study, we list the fundamental pillars of the large-scale computing infrastructures which enable its ODA capabilities, and conduct a study of the popular ODA frameworks operating in various such environments (predominantly HPC). Based on that, we propose a more holistic ODA framework matching the various layers of a large-scale graph-processing distributed ecosystem proposed by Sherif Sak et al, that extends the ODA functionalities presented in an existing novel ODA framework proposed by Netti et al. We compare the holistic ODA framework proposed by us to some of the state-of-the-art frameworks that we study as part of this literature to highlight the novelty, which would hopefully draw more attention to perform extensive research in this field. As part of creating awareness, we highlight the significant operational efficiencies observed as a result of the implementation of the state-of-the-art ODA frameworks to make the study appear beneficial for the readers, and lastly, discuss the trending research work ongoing in this field.