Advanced Python Performance Monitoring with Score-P

TL;DR

This paper presents a method to analyze and trace Python applications in high-performance computing environments using the Score-P instrumentation framework, addressing the lack of suitable tools for highly parallel Python programs.

Contribution

It demonstrates how to adapt the Score-P framework for Python performance analysis and evaluates the overhead involved in instrumenting Python applications.

Findings

Score-P can be effectively used for Python performance tracing.

The overhead of instrumentation is manageable for typical HPC Python applications.

The approach enables detailed performance insights for parallel Python programs.

Abstract

Within the last years, Python became more prominent in the scientific community and is now used for simulations, machine learning, and data analysis. All these tasks profit from additional compute power offered by parallelism and offloading. In the domain of High Performance Computing (HPC), we can look back to decades of experience exploiting different levels of parallelism on the core, node or inter-node level, as well as utilising accelerators. By using performance analysis tools to investigate all these levels of parallelism, we can tune applications for unprecedented performance. Unfortunately, standard Python performance analysis tools cannot cope with highly parallel programs. Since the development of such software is complex and error-prone, we demonstrate an easy-to-use solution based on an existing tool infrastructure for performance analysis. In this paper, we describe how to apply the established instrumentation framework \scorep to trace Python applications. We finish with a study of the overhead that users can expect for instrumenting their applications.