Parallel Astronomical Data Processing with Python: Recipes for multicore machines

TL;DR

This paper presents practical recipes for parallel data processing in astronomy using Python on multicore machines, demonstrating significant time savings across different processor types with easy-to-implement methods.

Contribution

It introduces accessible parallel processing recipes for astronomers using Python, comparing multiple multiprocessing approaches and providing benchmark results for varied complexity tasks.

Findings

Parallel processing significantly reduces execution time.

Python's multiprocessing module simplifies parallel implementation.

Different approaches have comparable performance benefits.

Abstract

High performance computing has been used in various fields of astrophysical research. But most of it is implemented on massively parallel systems (supercomputers) or graphical processing unit clusters. With the advent of multicore processors in the last decade, many serial software codes have been re-implemented in parallel mode to utilize the full potential of these processors. In this paper, we propose parallel processing recipes for multicore machines for astronomical data processing. The target audience are astronomers who are using Python as their preferred scripting language and who may be using PyRAF/IRAF for data processing. Three problems of varied complexity were benchmarked on three different types of multicore processors to demonstrate the benefits, in terms of execution time, of parallelizing data processing tasks. The native multiprocessing module available in Python makes it a relatively trivial task to implement the parallel code. We have also compared the three multiprocessing approaches - Pool/Map, Process/Queue, and Parallel Python. Our test codes are freely available and can be downloaded from our website.