pPython Performance Study

TL;DR

pPython is a Python-based parallel programming framework using PGAS semantics on top of a file-based messaging library, achieving comparable performance to mpi4py for large messages across various systems.

Contribution

This paper introduces pPython, a novel parallel programming model in Python utilizing file-based messaging and PGAS semantics, with performance analysis against mpi4py.

Findings

pPython achieves comparable performance to mpi4py for large messages.

pPython supports execution on single-node and heterogeneous systems.

Performance benchmarks include point-to-point and collective communications.

Abstract

pPython seeks to provide a parallel capability that provides good speed-up without sacrificing the ease of programming in Python by implementing partitioned global array semantics (PGAS) on top of a simple file-based messaging library (PythonMPI) in pure Python. pPython follows a SPMD (single program multiple data) model of computation. pPython runs on a single-node (e.g., a laptop) running Windows, Linux, or MacOS operating systems or on any combination of heterogeneous systems that support Python, including on a cluster through a Slurm scheduler interface so that pPython can be executed in a massively parallel computing environment. It is interesting to see what performance pPython can achieve compared to the traditional socket-based MPI communication because of its unique file-based messaging implementation. In this paper, we present the point-to-point and collective communication performances of pPython and compare them with those obtained by using mpi4py with OpenMPI. For large messages, pPython demonstrates comparable performance as compared to mpi4py.