A Framework for HI Spectral Source Finding Using Distributed-Memory Supercomputing

TL;DR

This paper introduces a scalable framework for spectral source finding in large radio astronomy datasets using distributed-memory supercomputing, enabling much faster processing than traditional methods.

Contribution

The work presents the Scalable Source Finder Framework and implements a Parallel Gaussian Source Finder, facilitating efficient analysis of petabyte-scale spectral data on supercomputers.

Findings

PGSF processed 256GB dataset in under 24 minutes

Framework supports various source finding algorithms

Significant reduction in processing time compared to traditional methods

Abstract

The latest generation of radio astronomy interferometers will conduct all sky surveys with data products consisting of petabytes of spectral line data. Traditional approaches to identifying and parameterising the astrophysical sources within this data will not scale to datasets of this magnitude, since the performance of workstations will not keep up with the real-time generation of data. For this reason, it is necessary to employ high performance computing systems consisting of a large number of processors connected by a high-bandwidth network. In order to make use of such supercomputers substantial modifications must be made to serial source finding code. To ease the transition, this work presents the Scalable Source Finder Framework, a framework providing storage access, networking communication and data composition functionality, which can support a wide range of source finding algorithms provided they can be applied to subsets of the entire image. Additionally, the Parallel Gaussian Source Finder was implemented using SSoFF, utilising Gaussian filters, thresholding, and local statistics. PGSF was able to search on a 256GB simulated dataset in under 24 minutes, significantly less than the 8 to 12 hour observation that would generate such a dataset.