Driving Computational Efficiency in Large-Scale Platforms using HPC Technologies

TL;DR

This paper analyzes HPC resource utilization in the LAGO astroparticle physics project, identifying inefficiencies and providing data-driven recommendations to optimize computational throughput and scientific productivity.

Contribution

It offers a detailed analysis of resource consumption patterns in LAGO's HPC environment and proposes strategies for improving efficiency based on workload characterization.

Findings

High CPU efficiency in simulation tasks

Short test jobs distort aggregate efficiency metrics

Recommendations for optimizing resource requests and workflows

Abstract

The Latin American Giant Observatory (LAGO) project utilizes extensive High-Performance Computing (HPC) resources for complex astroparticle physics simulations, making resource efficiency critical for scientific productivity and sustainability. This article presents a detailed analysis focused on quantifying and improving HPC resource utilization efficiency specifically within the LAGO computational environment. The core objective is to understand how LAGO's distinct computational workloads-characterized by a prevalent coarse-grained, task-parallel execution model-consume resources in practice. To achieve this, we analyze historical job accounting data from the EGI FedCloud platform, identifying primary workload categories (Monte Carlo simulations, data processing, user analysis/testing) and evaluating their performance using key efficiency metrics (CPU utilization, walltime utilization, and I/O patterns). Our analysis reveals significant patterns, including high CPU efficiency within individual simulation tasks contrasted with the distorting impact of short test jobs on aggregate metrics. This work pinpoints specific inefficiencies and provides data-driven insights into LAGO's HPC usage. The findings directly inform recommendations for optimizing resource requests, refining workflow management strategies, and guiding future efforts to enhance computational throughput, ultimately maximizing the scientific return from LAGO's HPC investments.