Scalability Model for the LOFAR Direction Independent Pipeline

TL;DR

This paper develops a scalability model for the LOFAR DI pipeline, enabling prediction and optimization of processing times for large radio astronomy data sets, thereby improving efficiency for large surveys.

Contribution

The paper presents a comprehensive scalability model for the LOFAR prefactor pipeline, incorporating parameters like CPU count and data size, to optimize processing of large astronomical datasets.

Findings

Smaller calibration models significantly reduce calibration time.

The model accurately predicts processing times across various parameters.

Shared cluster processing incurs measurable performance penalties.

Abstract

LOFAR is a leading aperture synthesis telescope operated in the Netherlands with stations across Europe. The LOFAR Two-meter Sky Survey (LoTSS) will produce more than 3000 14 TB data sets, mapping the entire northern sky at low frequencies. The data produced by this survey is important for understanding the formation and evolution of galaxies, supermassive black holes and other astronomical phenomena. All of the LoTSS data needs to be processed by the LOFAR Direction Independent (DI) pipeline, prefactor. Understanding the performance of this pipeline is important when trying to optimize the throughput for large projects, such as LoTSS and other deep surveys. Making a model of its completion time will enable us to predict the time taken to process large data sets, optimize our parameter choices, help schedule other LOFAR processing services, and predict processing time for future large radio telescopes. We tested the prefactor pipeline by scaling several parameters, notably number of CPUs, data size and size of calibration sky model. We present these results as a comprehensive model which will be used to predict processing time for a wide range of processing parameters. We also discover that smaller calibration models lead to significantly faster calibration times, while the calibration results do not significantly degrade in quality. Finally, we validate the model and compare predictions with production runs from the past six months, quantifying the performance penalties incurred by processing on a shared cluster. We conclude by noting the utility of the results and model for the LoTSS Survey, LOFAR as a whole and for other telescopes.

Tables

Table 1. Table 1 : Averaging parameters and final data sizes tested for the sample LOFAR SKSP observation. The raw data is 64 GB per subband. The LOFAR SKSP data processing uses averaging parameters of 8 seconds and 2 channels per subband. This reduces the raw data by a factor of 64. We highlight the data size used in the LOFAR SKSP survey.

Averaging
ratio	Time averaging
	Channels per Subband	Averaged
		Size (Gb)
64x	8	2	1.235
32x	4	2	2.459
16x	2	2	4.906
8x	1	2	9.802
4x	1	4	18.00
2x	1	8	36.72
1x	1	16	66.88

Table 2. Table 2 : List of test sky models. Model 3 is created with the parameters used in our production processing of LOFAR data. All models include objects within 5 degrees from the centre of the pointing.

Sky model #	min sensitivity	# sources
model 1	0.05 Jy	809
model 2	0.1 Jy	503
model 3	0.3 Jy	180
model 4	0.5 Jy	96
model 5	0.8 Jy	49
model 6	1.0 Jy	34
model 7	1.5 Jy	16

Table 3. Table 4 : Statistics for an empty region for the four images shown in Figure 7 . The 0.3Jy model, here shown shaded in gray, is the one used in production.

Calibration Model Flux Cutoff	# of sources	RMS Noise (Jy)
0.05Jy	809	0.00402834
0.3 Jy	180	0.00402311
0.8 Jy	49	0.00404181
1.5 Jy	16	0.00410204

Table 4. Table 7 : Statistics for queuing time for different values of CPUs requested. Queueing time for jobs requesting less than 8 CPUs is typically less than five minutes. It can drastically increase for larger jobs.

NCPU requested	Mean time (sec)	Median time (sec)	75th percentile (sec)
1 CPU	150.5	116.2	154.1
2 CPU	201.1	125.8	165.8
3 CPU	296.2	152.0	243.0
4 CPU	498.9	167.7	233.7
8 CPU	1944.2	428.4	2142.4
16 CPU	7079.0	696.4	8750.6

Table 5. Table 11 : Fit parameters for the models in Equation 1 .

prefactor step	$R^2$	P value	Standard Error
predict_ateam	0.996	0	$1.92\times {10}^{-10}$
ateamcliptar	0.979	0	$3.94\times {10}^{-11}$
dpppconcat	0.999	$1.2\times {10}^{-128}$	$1.78\times {10}^{-10}$
gsmcal_solve 16GB	0.995	$3.12\times {10}^{-75}$	$6.80\times {10}^{-9}$
gsmcal_solve $>$16GB	0.951	$7.07\times {10}^{-40}$	$1.58\times {10}^{-8}$
gsmcal_apply	0.989	$5.6\times {10}^{-82}$	$3.12\times {10}^{-10}$

Table 6. Table 14 : Goodness of fit parameters for the model in Equation 4 . Since the model is split into two parts, we treat each section as a single linear model.

Value of $𝒩$	$R^2$	P value	Standard Error
$𝒩\le 4$	0.382	0.381	37.086
$𝒩>4$	0.986	0.075	86.293

Equations

$$T_{predict\_ateam}=5.19\times 10^{-8}\mathcal{S}+4.20\times 10^1$$

$$T_{ateamcliptar}=4.57\times 10^{-9}\mathcal{S}-8.42\times 10^0$$

$$T_{dpppconcat}=3.51\times 10^{-8}\mathcal{S}+4.20\times 10^1$$

$$T_{gsmcal\_solve}=\{\begin{array}{ll}7.38\times 10^{-7}\mathcal{S}-8.20\times 10^1& |\mathcal{S}\le 1.6\times 10^{10}\\ 1.04\times 10^{-6}\mathcal{S}-4.04\times 10^3& |\mathcal{S}>1.6\times 10^{10}\end{array}$$

$$T_{gsmcal\_apply}=2.07\times 10^{-8}\mathcal{S}-1.38\times 10^1$$

$$T=1185\cdot {\mathcal{F}}^{-0.854}$$

$$T=503.37+\frac{3062.6}{\mathcal{N}}$$

$$T=\{\begin{array}{ll}49.3\cdot \mathcal{N}+120& |\mathcal{N}\le 4\\ 726\cdot \mathcal{N}-3071& |\mathcal{N}>4\end{array}$$

$$T=5.918\times 10^{20}\cdot {\mathcal{S}}^{2.336}$$

$$\begin{array}{rl}{t}_{infrastructure}& =\{\begin{array}{ll}49.3\cdot \mathcal{N}+120& |\mathcal{N}\le 4\\ 726\cdot \mathcal{N}-3071& |\mathcal{N}>4\end{array}\\ & +2\cdot 0.056\cdot {\mathcal{S}}^{2.336}\end{array}$$

$$\begin{array}{rl}{t}_{gsmcal\_solve}& =t_{infrastructure}\\ & +[3566\cdot \frac{1}{3.012}{\mathcal{F}}^{-0.854}\cdot (0.1412+\frac{0.8589}{\mathcal{N}})]\cdot \{\begin{array}{l}7.38\cdot 10^{-7}\mathcal{S}|\mathcal{S}\le 16\\ 1.04\cdot 10^{-6}\mathcal{S}|\mathcal{S}>16\end{array}\end{array}$$

$$\begin{array}{rl}{t}_{dpppconcat}& =t_{infrastructure}\\ & +3.51\times 10^{-8}\mathcal{S}+4.20\times 10^1\end{array}$$

$$\begin{array}{rl}{t}_{predict\_ateam}& =t_{infrastructure}\\ & +5.19\times 10^{-8}\mathcal{S}+4.20\times 10^1\end{array}$$

$$\left[\begin{array}{cc}6.83\times 10^2& -1.94\times 10^{-1}\\ -1.94\times 10^{-1}& 5.81\times 10^{-5}\end{array}\right]$$

$$\left[\begin{array}{cc}171.94& -504.11\\ -504.11& 2322.95\end{array}\right]$$

$$\left[\begin{array}{cc}2.53\times 10^{-4}& 1.08\times 10^{-3}\\ 1.08\times 10^{-3}& 4.60\times 10^{-3}\end{array}\right]$$

Full text

††journal: Astronomy And Computing

\DeclareCaptionType

equ[Equation][List of Equations]

1 Introduction

Astronomy has entered the big data era with many projects creating petabytes of data per year. This data is often processed by complex multi-step pipelines consisting of various algorithms. Understanding the scalability of astronomical algorithms theoretically, in a controlled environment, and in production is important for making predictions for the data reduction of future projects and upcoming telescopes.

The Low Frequency Array (LOFAR) (Van Haarlem et al., 2013) is a leading European low-frequency radio telescope. The majority of LOFAR’s stations are in the Netherlands, however the telescope can use stations across Europe to create ultra-high resolution radio maps. LOFAR data needs to undergo several computationally intensive processing steps before obtaining a final scientific image.

To create a broadband image, LOFAR data is first processed by a Direction Independent (DI) Calibration pipeline followed by Direction Dependent (DD) Calibration pipeline (e.g. Van Weeren et al., 2016, Williams et al., 2016, Smirnov and Tasse, 2015, Tasse et al., 2018). The goal of DI calibration is to remove effects that are constant across the target field such as radio frequency interference, contamination by bright off-axis sources and antenna gains. After this step, DD Calibration focuses on removing effects which vary across the field, such as ionospheric and beam effects. The result of these two pipelines is a science-ready image.

Our implementation of the DI LOFAR processing, prefactor, can be parallelized on a high throughput cluster (Mechev et al., 2017). The Direction Dependent processing, implemented in ddf-pipeline111Available at https://github.com/mhardcastle/ddf-pipeline/releases, is subsequently performed on a single HPC node.

The LOFAR Surveys Key Science Project (SKSP) (Shimwell et al., 2017, 2018) is a long running project consisting of several low frequency surveys of the northern sky. The broadest tier of the survey, LoTSS, will use more than 3000 8-hour observations to create maps with a noise levels below 100 $\mu$Jy. We have already processed more than 500 of these observations using the prefactor DI pipeline (Van Weeren et al., 2016, Horneffer et al., 2018).

While the current LoTSS imaging algorithms can process data averaged by up to a factor of 64 in frequency and time, it is important to understand how LOFAR processing scales with processing parameters, such as averaging parameters. Since LOFAR data is used by multiple scientific teams, not every team can produce scientific results from data averaged by such a high factor. Users from those teams need to be able to predict the time and computational resources required to process their data, taking into account the increasing LOFAR observation rates, data sizes and scientific requirements.

We study the scalability of processing LOFAR data, by setting up processing of a sample SKSP data set on an isolated node on the GINA cluster at SURFsara, part of the Dutch national e-infrastructure (Templon and Bot, 2016). We test the software performance as a function of several parameters, including averaging parameters, number of CPUs and calibration model size. Additionally, we test the performance of the underlying infrastructure, i.e. queuing and download time, for the same parameters. Finally, we compare those isolated tests with our production runs of the prefactor pipeline to measure the overhead incurred by running on a shared system.

We discover that the computationally intensive LOFAR processing steps scale linearly with data size, and calibration model size. Additionally, we find that the time taken by these steps is inversely proportional to the number of CPUs used. We discover that the time to download and extract data on the GINA cluster is linear with size up to 32GB, but becomes slower beyond this data size. We also find that the queuing time on the GINA cluster grows exponentially for jobs requesting more than 8 CPUs. We validate these isolated tests with production runs of LOFAR data from the past six months. We combine all these tests into a single model and show its prediction power by testing the processing time for different combinations of parameters. Finally, we discuss the utility of our method, the results in this work and applications to the SKSP projects, the broader impact of our results to LOFAR processing and the applications for other large astronomical surveys. The major contributions of this work can be summarized as:

A model of processing time for the LOFAR Direction Independent Calibration Pipeline.

• 2.

A model of queueing time and file transfer time which is used by current or future jobs processed on the GINA cluster.

• 3.

Quantification of overheads incurred when processing in production.

• 4.

Validation of our methods with discussion of future applications.

We introduce LOFAR processing and other related work in Section 2 and describe our software setup and data processing methods in Section 3. We present our results and performance model in Section 4 and discussions and conclusions in Section 5.

2 Related Work

In previous work, we have parallelized the Direction Independent LOFAR pipeline on a High Throughput infrastructure (Mechev et al., 2017). While this parallelization has helped accelerate data processing for the SKSP project, creating a performance model of our software is required if we are to predict the resources taken by future jobs. This model will be particularly useful in understanding how processing parameters will affect run time.

In previous work, we have parallelized the Direction Independent LOFAR pipeline on a High Throughput infrastructure (Mechev et al., 2017). While this parallelization has helped accelerate data processing for the SKSP project, creating a performance model of our software is required if we are to predict the resources taken by future jobs. This model will be particularly useful in understanding how processing parameters will affect run time.

Performance modelling on a distributed system is an important field of study related to grid computing. A good model of the performance of tasks in distributed workflows can help more efficiently schedule these jobs on a grid environment (Sanjay and Vadhiyar, 2008). The performance modeling systems require knowledge of the source code and an analytical model of the slowest parts of the code (Xu et al., 1996). Many systems exist to model the performance of distributed jobs (Barnes et al., 2008, Xu et al., 1996, Kuperberg et al., 2008, Witt et al., 2018), with some employing Black Box testing (Yang et al., 2005, Kavulya et al., 2010) or tests on scientific benchmark cases (Carrington et al., 2006). Such performance analysis does not require intimate knowledge of the software and can be applied on data obtained from processing on a grid infrastructure.

Empirical modelling is useful in finding performance bugs in parallel code (Calotoiu et al., 2013) and modelling the performance of big data architectures (Castiglione et al., 2014). The insights from these models are used to optimize the architecture of the software system or determine bottlenecks in processing. Here, we use empirical modelling to determine how the LOFAR prefactor performance scales with different parameters.

3 Processing Setup

Using the LOFAR software installation described in Mechev et al. (2017), we processed a typical LOFAR SKSP observation222LOFAR Observation ID L658492, co-ordinates [17h42m21.785, +037d41m46.805] observed by the LOFAR High Band Array for 8 hours between 2018-06-20 and 2018-06-21., while changing the averaging rate in time and frequency. Changing these averaging parameters will change the final data size (with the data sizes studied shown in Table 3.1). We test the processing time for different averaging parameters by running 15 runs per parameter step.

The data used by the LOFAR surveys is archived at a time resolution of 1 second intervals and frequency resolution of 16 channels per subband (equivalent to 12kHz channel width). While some of the processing steps such as flagging of Radio Frequency Interference and removal of bright off-axis sources produce better results when performed on the high-resolution data, later steps can be performed on averaged data with little impact on the final product quality. To speed up processing, the raw data is averaged in time and frequency, decreasing the input data size to later tasks. The main aims of the LoTSS survey can be accomplished if the final data products from the prefactor pipeline are averaged to a time resolution of 8 seconds per sample and frequency of 2 channels per subband. These averaging parameters correspond to a reduction in data size by a factor of 64. Nevertheless, other science cases require less averaging of the data. Our aim is to understand how the processing of this larger data will scale. To measure the scalability of processing, we measure the performance of the prefactor pipeline for data sizes between the raw data of 64GB/subband and the averaged data of 1GB/subband. The tested data sizes and parameters are shown in table 3.1, and discussed in Section 4.1.1.

We performed the scalability tests on a dedicated node of the SURFsara GINA cluster, f18-01. The node is a typical hardware node used by our production LoTSS processing, however it is dedicated for the tests in order to ensure there is no contamination by other software. The node is described in Section 3.3.

We processed the sample data set with the LOFAR prefactor pipeline. The prefactor version used was the same as we use for the LOFAR SKSP broadband surveys (Horneffer et al., 2018). This software consists of several steps executed in sequence, shown graphically in Figure 1. The important prefactor steps are as follows. The predict_ateam and ateamcliptar steps predict the contamination by bright off-axis sources and remove these effects respectively. The dpppconcat step is responsible for concatenating 10 subbands into a single file which is in turn calibrated. The step gsmcal_solve is responsible for calibration of the data against a model of the radio sky. The solutions produced by gsmcal_solve are used by gsmcal_apply and applied to the scientific observation.

3.1 Processing Metrics

The goal for our scalability model is to understand the effect of several parameters on the job completion time of LOFAR software. We do this by testing the processing time for various values of data size, number of CPUs used and sky model size.