The LOFAR Known Pulsar Data Pipeline

TL;DR

The paper describes the development of the LOFAR Known Pulsar Data Pipeline, a high-capacity, scalable system designed for processing large-scale pulsar data streams using advanced computational infrastructure and data formats.

Contribution

It introduces a new pulsar data pipeline leveraging supercomputing resources and HDF5 format for efficient large-scale radio astronomy data processing.

Findings

Pipeline successfully processes up to 23TB/hour of data

Uses HDF5 for efficient storage of complex data

Lays groundwork for next-generation pulsar search tools

Abstract

Transient radio phenomena and pulsars are one of six LOFAR Key Science Projects (KSPs). As part of the Transients KSP, the Pulsar Working Group (PWG) has been developing the LOFAR Pulsar Data Pipelines to both study known pulsars as well as search for new ones. The pipelines are being developed for the Blue Gene/P (BG/P) supercomputer and a large Linux cluster in order to utilize enormous amounts of computational capabilities (50Tflops) to process data streams of up to 23TB/hour. The LOFAR pipeline output will be using the Hierarchical Data Format 5 (HDF5) to efficiently store large amounts of numerical data, and to manage complex data encompassing a variety of data types, across distributed storage and processing architectures. We present the LOFAR Known Pulsar Data Pipeline overview, the pulsar beam-formed data format, the status of the pipeline processing as well as our future plans for developing the LOFAR Pulsar Search Pipeline. These LOFAR pipelines and software tools are being developed as the next generation toolset for pulsar processing in Radio Astronomy.