Development details and computational benchmarking of DEPAM

TL;DR

This paper introduces a scalable, cloud-based system for FFT-based acoustic data analysis in oceanography, demonstrating promising performance and scalability compared to existing tools.

Contribution

We developed and benchmarked a Hadoop/Spark system for acoustic feature extraction, showing improved scalability and performance over traditional methods.

Findings

System performs well on single-node setups

System scales almost linearly with dataset size

Benchmarking shows competitive performance with existing tools

Abstract

In the big data era of observational oceanography, passive acoustics datasets are becoming too high volume to be processed on local computers due to their processor and memory limitations. As a result there is a current need for our community to turn to cloud-based distributed computing. We present a scalable computing system for FFT (Fast Fourier Transform)-based features (e.g., Power Spectral Density) based on the Apache distributed frameworks Hadoop and Spark. These features are at the core of many different types of acoustic analysis where the need of processing data at scale with speed is evident, e.g. serving as long-term averaged learning representations of soundscapes to identify periods of acoustic interest. In addition to provide a complete description of our system implementation, we also performed a computational benchmark comparing our system to three other Scala-only, Matlab and Python based systems in standalone executions, and evaluated its scalability using the speed up metric. Our current results are very promising in terms of computational performance, as we show that our proposed Hadoop/Spark system performs reasonably well on a single node setup comparatively to state-of-the-art processing tools used by the PAM community, and that it could also fully leverage more intensive cluster resources with a almost-linear scalability behaviour above a certain dataset volume.

Figures

Tables

Table 1. Table 2.1: Parameter sets of the FFT-related variables in the workflow: nfft (number of points in FFT), windowOverlap (number of overlapping samples in consecutive windows), windowSize (number of samples of short-term analysis windows), recordSizeInSec (number of samples of longer time segments over which periodograms are averaged). At the line Processing load we report the : number of integration segments per audio file / volume (in MB) of each segment / number of analysis windows per segment.

Parameter	Set 1	Set 2
nfft	256	1024
windowOverlap	128	0
windowSize	256	1024
recordSizeInSec	1	30
Processing load	2700 / 0.06 / 691213	90 / 1.92 / 86401

Table 2. Table 2.2: Nominal configurations of the tested computing systems. The parallel frameworks are used in a single node setup, while the distributed framework is used for a multiple node setup (only tested for SparkSys).

System	Language (version)	Parallel framework	Distributed framework
SparkSys / SparkSysOpti	Scala (2.11.8)	Pseudo-distributed Spark	Hadoop / Spark
ScalaSys	Scala (2.11.8)	Standard parallel collections	-
MatlabSys	Matlab (2016b)	Parallel Computing Toolbox	-
PythonSys	Python (3.5)	Scipy/Multiprocessing	-

Full text

• •

Joseph Allemandou3

• •

Dorian Cazau1

• •

Alexandre Degurse4

• •

Paul Nguyen Hong Duc2

belonging to the following institutes (at the time of their contribution)

Lab-STICC, Institute Mines Telecom Atlantique 2. 2.

Institut Jean le Rond d’Alembert, Sorbonne Universités 3. 3.

JoalTech 4. 4.

ENSTA Bretagne

• •

Nguyen, P., Degurse, A., Allemandou, J., Adam, O., Gerard, O., White, P. R., Fablet, R. and Cazau, D. (2019) ”A scalable Hadoop/Spark framework for general-purpose analysis of high volume passive acoustic data”, IEEE OCEANS 2019, Marseille, June 17-20.

1. 1 Introduction

2. 1.1 Context

3. 1.2 Contributions

4. 2 Methods

5. 2.1 Proposed Hadoop/Spark-based system

6. 2.1.1 System overview

7. 2.1.2 Implementation details

8. 2.2 Experimental setup

9. 2.2.1 Infrastructure

10. 2.2.2 Workflow & dataset

11. 2.2.3 Tested systems

12. 2.2.4 Evaluation

13. 3 Results & Discussion

14. 3.1 Single node experimentation

15. 3.1.1 Benchmarking of systems

16. 3.1.2 Sensitivity studies of systems

17. 3.2 Multiple node experimentation

18. 4 General discussion

Chapter 1 Introduction

1.1 Context

Technological progress in observational oceanography gave rise to a two-tiered system in which major strategic investments have been put primarily in data acquisition rather than in data management and processing plans. As a result, there is currently a huge gap between in-situ small-scale data acquisition and a more integrated global knowledge that could be directly used in operational oceanography research and by decision-making managers. A good example of scientific community facing these difficulties is the underwater Passive Acoustic Monitoring (PAM) one, which investigates biological (e.g, whale census) and human (e.g., ship noise monitoring) activities, as well as physical processes (e.g, wind speed and rainfall estimation), in the ocean. Specifically, due to the development of cabled observatories that now provide virtually unlimited power for high bandwidth, continuous data acquisition, and the increase of storage capacity and life battery of temporary recorders, the volume of datasets to process has become larger and larger. For instance, the PerenniAL Acoustic Observatory in the Antarctic Ocean (PALAOA) observatory has been recording quasi-continuously the underwater soundscape of the Southern Ocean since 2005 Boebel et al. (2006), generating about 140 GB per day Kindermann et al. (2008), and the Ocean Network Canada has collected more than 300 TB of PAM data in their database Biffard et al. (2018). In France, governmental agencies like Service Hydrographique et Océanographique de la Marine (SHOM) and Agence Française de la Biodiversité (AFB) are also experiencing similar challenges of processing large volume of data in the Directive Cadre Stratégie pour le Milieu Marin (DCSMM) context, where anthropogenic ambient noise analysis and marine mammal census have to be performed on a long-term continuous effort.

Several projects have started to address the question of processing high volume PAM data more efficiently by adopting distributed computing systems. A distributed computing system can be simply defined as a “system whose components are located on different networked computers (or nodes), which communicate and coordinate their actions by passing messages to one another” 111From https://en.wikipedia.org/wiki/Distributed_computing.. Each computer has its own multiprocessor structure and memory. This makes it good for redundant storage and availability, durability. In contrast, local systems based on a single node, as usually used in the PAM community, all processors may have access to a shared memory to exchange information between processors, like when performing multiprocessor parallel computing. Among well-known distributed environments within the big data space, Apache Spark has become a prominent player. Initially developed in 2012 at the AMPLab at UC Berkeley, Spark is an “open-source distributed general-purpose cluster-computing framework, also providing an interface for programming entire clusters with implicit data parallelism and fault tolerance” 222From https://en.wikipedia.org/wiki/Apache_Spark..

Big data analytics in the PAM community is only in its early stages. Spark Streaming has already been developed for routine processing (filtering and spectrogram analysis) for large terrestrial datasets Thudumu et al. (2016). Using Hadoop Distributed File System (HDFS) as a distributed storage system, their system resulted in better runtime performance in comparison to standalone execution, with approximately 78% reduction in the execution time. Concurrent computational approaches have used the Matlab Parallel Computing Toolbox and Matlab Distributed Computing Server to run detection and classification algorithms for whale species recognition Dugan et al. (2016). Their most improved process (classifier-based detection) was 6.57× faster for an 8-node server over a serial process. Other more bespoke master-slave models with data distribution have also been developed, where a master node first splits large audio files into smaller chunks, creates a list of work tasks that are distributed over the nodes and eventually aggregates each node output. Such a system has been developed for acoustic event detection and bioacoustic spectral indices Truskinger et al. (2014), improving average execution time by 24x for a 5 instance, 32 thread distributed cluster over a single threaded process. Using a similar approach optimized for scalability, a pipeline of different preprocessing operations to reduce concurrent noise sources in audio recordings, is 21.76 times faster with 32 cores over 8 virtual machines, compared to a serial process Brown et al. (2018).

1.2 Contributions

In this paper, we wish to share these efforts by proposing a scalable computation chain for FFT (Fast Fourier Transform)-based features based on the Hadoop and Spark frameworks. These features (e.g., full frequency band Sound Pressure Levels, SPL) are at the core of many different types of acoustic analysis where the need of processing data at scale with speed is evident. For example, these features often serve as long-term averaged learning representations (among them, the well-known Long-Term Spectral Average, LTSA) of soundscapes to identify periods of acoustic interest Erbe et al. (2015); Merchant et al. (2015), which is done either manually or with image-based pattern recognition methods Frasier et al. (2018). Such applications, namely fast automatic content report and interactive annotation of large datasets, need fast and scalable computations of the features to be performed efficiently. Furthermore, LTSA generation relies on several processing parameters (e.g., analysis window size) that can highly modify event-specific averaged patterns and reduce the interpretability of LTSA Hawkins et al. (2014). To better assess this variability, systematic comparative testing of different parameter sets need to be carried out, which also requires intensive computing.

In addition to develop a new scalable Hadoop/Spark system for high performance computing of FFT-features, we also provide a computational benchmark comparing our system against three other computing systems based on different programming languages (Scala, Matlab and Python) in standalone executions, using execution time as evaluation metric. We also evaluate the system scalability in its distributed configuration.

Chapter 2 Methods

2.1 Proposed Hadoop/Spark-based system

Our proposed distributed computing system, based on the Apache Hadoop and Spark frameworks, is shown in Figure 2.1.

2.1.1 System overview

Hadoop is responsible for distributed data storage and resource management (including job scheduling/monitoring) accross multiple nodes of a cluster, relying respectively on Hadoop Distributed File System (HDFS) and Yet Another Resource Negotiator (YARN). The main function of HDFS is to divide a data file (e.g. 45-mins long / 169 MB for audio files) into smaller blocks of specified size (default is 64 MB). Each block is processed by one map process and map processes run in parallel. In HDFS, the master is the NameNode which manages the filesystem namespace and logs all modifications and the state of the filesystem. It communicates all the information about the content of a filesystem to the DataNodes, corresponding to different machines where the HDFS blocks are locally located. Regarding YARN, the Application Master (here the Spark Driver as we will see after) negotiates resources with the ResourceManager, which is responsible for granting containers corresponding to the resources allocated (container Hadoop = N cores + M GB RAM). Containers are then supervised locally by the NodeManagers. Globally, as represented by the dashed arrows in figure 2.1, both Hadoop components HDFS and YARN communicate with other machines through a master-slave model as follows: NameNode $\leftrightarrow$ DataNodes for HDFS and ResourceManager $\leftrightarrow$ NodeManagers for YARN.

Spark focuses on processing data in parallel across a cluster. When used in conjunction with Hadoop, the Spark Driver organizes the completion of the jobs across the cluster of executors by interacting with the ResourceManager and the NodeManagers. Jobs are performed across the worker nodes (CPUs (cores) and allocated memory) using Stages and Tasks. Two main components of the Spark architecture are:

• •

Spark Driver: this is the Application Master in our workflow. It tracks all the operations by executors. Moreover, this entity parses the code, and serializes the byte level code across the executors. Any computation is actually done at the local level by each of them. Furthermore, the Driver aims to plan all the computation in the cluster with Directed Acyclic Graph (DAG). Once a DAG is created, it represents a job which is divided in stages. Then, each stage is carried out as tasks. Finally, the Driver handles fault tolerance of all performed operations;

• •

Spark Executors: represent processes running in the containers in a cluster. One or more executors could be in each worker node and multiple tasks can be run in a single executor.

2.1.2 Implementation details

The three parameters num-executors (number of executors requested), executor-cores (number of tasks the executor can run simultaneously), and executor-memory (controlling the executor heap size) play a key role in performance of the Spark system as they control the amount of CPU and memory the application gets. The parameters of the nominal configuration, called SparkSys, have been set as follows: executor-cores=3, num-executors=8 and executor-memory=11.5GB. A more optimized version, which corresponds roughly to the best parameter setup in the balance between system performance and resource allocation, and called SparkSysOpti, have also been tested, with parameters: executor-cores=3, num-executors=17 and executor-memory=5.5GB. In particular, it was observed that HDFS client has trouble with a great number of concurrent threads, and that it achieves full write throughput with 5 or less tasks per executor.

When using N node clusters, one node is used as a master and remaining N-1 as slaves. Indeed, when a Spark application is run using a cluster manager like YARN, several daemons will run in the background like NameNode, Secondary NameNode, DataNode, JobTracker and TaskTracker. Thus, while specifying num-executors, we need to make sure that we leave aside enough cores (typically 1 core per node) for these daemons to run smoothly. Furthermore, the programming language Scala (version 2.11.8) has been used to implement the tested workflow (described in Sec. 2.2.2), and we also used the multithreaded FFT library JTransforms 111https://github.com/wendykierp/JTransforms, written in Java. Note that Scala is a programming language that has flexible syntax as compared to other languages like Python or Java, and Apache Spark itself is written in Scala as it is more scalable on Java Virtual Machine.

2.2 Experimental setup

2.2.1 Infrastructure

All our numerical experimentations have been performed on the DATARMOR infrastructure (http://www.ifremer.fr/pcdm), belonging to IFREMER. Each node is composed of an Intel Xeon 2X CPU E5-2680 v4 (28c / 56t), 128 GB DDR3, i.e. up to 56 cores (28 $\times$ 2 hyperthreaded CPUs) and 128 GB RAM per node. The multi-node Hadoop-Spark cluster of the SparkSys system was also deployed on DATARMOR. Up to 16 nodes were used to test different distributed configurations o this system. Each node of the cluster runs recent versions of Hadoop and Spark, i.e. 2.8 and 2.4.0, respectively, within the SUSE environment.

Note that the infrastructure architecture of Datarmor may be under-optimal for computational performance based on Hadoop/Spark frameworks, especially for high volume data, as each node does not have its own hard drive, making the data pass through different pipes with limited I/O throughput instead of being read and written locally. Furthermore, there is no dedicated access to node resources, which are instead shared among users (especially in terms of I/O throughput).

2.2.2 Workflow & dataset

The workflow used for the FFT-based feature computations is based on classical PAM analysis blocks (see e.g. Merchant et al. (2015) for background information), including three main steps: short-term FFT analysis (e.g., 32 ms), feature computations and feature integration over longer time segments (e.g., 1 min). Three FFT-based features have been computed: pwelch spectra, Third Octave Levels (TOL) and Sound Pressure Levels (SPL). It is noteworthy that we performed two independent segmentations at different time resolutions to cope with the minimal time window expected in TOL features, set to 1s according to ISO and ANSI standards ISO (1975); ANSI (2009). Also, all the results were sorted by time order and saved in JSON files. A complete description of this workflow, including both theory and implementation details, is available OSmOSE (2019). The two parameter sets used for our experiments are listed in table 2.1.

The dataset used to evaluate computational performance of computing systems is a real underwater PAM dataset recorded at 32,768 Hz near the archipelago of Saint-Pierre-et-Miquelon over the last three months of the year 2010. It consists of 1807 45-min long wav files for a total volume of 320 GB, each file being 169 MB.

2.2.3 Tested systems

Table 2.2 describes the three different computing systems tested. SparkSys has already been described in Sec. 2.1. It was benchmarked for computational performance against three other systems based on the Scala (v 2.11.8), Matlab (v 2016b) and Python (v 3.5) programming languages, respectively called ScalaSys, MatlabSys and PythonSys.

For all systems, we tried at best to comply with some “best practices in programming”, drawing from template-like codes that are widely used in the PAM community (e.g., the PAMGuide toolbox by Merchant et al. (2015)) for the Matlab implementation, and in the data scientist communities (e.g., the Scipy toolbox, https://www.scipy.org/) for the Scala and Python implementations. Double-precision floating-point format has been used in all three implementations. Multiple unitary tests have been performed on the core features of the workflow, and the outputs were cross-validated with a root mean square error below 10*-16* 222Cross-validation tests can be reproduced following our codes here https://github.com/Project-ODE/FeatureEngine-benchmark/blob/master/run-tests.sh.

In their nominal configurations, ScalaSys is the exact same Scala code as in SparkSys but without the Spark and Hadoop connections, using instead parallel collections included in the Scala standard library for multi-threaded processing, set with a default value of 24 threads. MatlabSys runs parallel Matlab code on a single node using the Parallel Computing Toolbox with 24 workers (equivalent to threads). Due to logistic reasons, we have not tested the use of Matlab Distributed Computing Server in a multiple node setup. Also, on a single node, PythonSys uses Scipy as backend to compute acoustic features, and the Multiprocessing library to run computations in parallel. A distributed version of PythonSys based on the Dask library is being developed and will be reported in future publications.

2.2.4 Evaluation

To evaluate computational performance of our different systems, execution time was assessed. We paid attention that software launch was not included in the execution time computation. Two different types of experimentations have been run, corresponding either to a parallel (single node) or to a distributed (multiple node) setup, as described in the following. We have also performed sensitivity studies for each system individually, which allows to assess performance variability around the nominal configurations given by table 2.2. To determine fluctuation in execution times, each tested system configuration has been executed 3 times, then the execution times were averaged over these 3 executions, and the standard deviations were computed.

Single node experimentation

As Spark also supports a pseudo-distributed local mode, we first benchmarked the three computing systems (see table 2.2) executed in a single node mode, over a linear increase of workloads, from 0.169 GB (1 wav file) to 16.9 GB (100 wav files).

Multi-node experimentation

In a second experimentation, we evaluated the scalability of our Hadoop/Spark system using the speed up metric (also referred to as improvement rate in Brown et al. (2018)), which corresponds to the reduction of execution time due to running a fixed workload using an increased number of hardware processors. A linear increase of workloads, from 16.9 GB (50 wav files) to 270.4 GB (1600 wav files), has been used. Note that, comparatively, workloads used for scalability analysis in the literature appear to be very small, e.g. in Thudumu et al. (2016) and Brown et al. (2018) data volumes of less than 5 GB are used. The other systems have not been tested in their distributed configurations.

Chapter 3 Results & Discussion

3.1 Single node experimentation

3.1.1 Benchmarking of systems

Figures 3.1 and 3.2 compare computational performance of the different computing systems in a single node setup, representing execution time (in mins) against workload (GB) for parameter sets 1 (on the left) and 2 (on the right). SparkSysOpti and MatlabSys perform quite similarly, especially for parameter set 1, both largely outperforming ScalaSys and SparkSys, and outperformed by PythonSys. For example, for parameter set 1 and a workload of 16.9 GB (i.e. 100 wav files of our dataset), it takes 2.5 minutes of computation time for SparkSysOpti and MatlabSys, which is more than twice as faster than ScalaSys and SparkSys but more than twice as slower than PythonSys. The difference between SparkSysOpti and MatlabSys is a bit more pronounced on the second set of parameters, although the performance slopes remain quite similar, while PythonSys still outperforms them. Standard deviation values of execution times are relatively minor comparatively to computational gain: 4 s ($\pm$ 3.2) for SparkSys, 2.5 s ($\pm$ 2.2) for ScalaSys and 8 s ($\pm$ 4.8) for MatlabSys.

In definitive, SparkSys performs reasonably well in its standalone mode, with performance similar to the Scala-only version. The slight optimization of its core parameters done with SparkSysOpt allows it to scale up easily and reach MatlabSys performance. This result is particularly interesting as it reveals, although the expected advantage of Apache Spark technology is to scale out processing over several nodes, that our system is also valuable on a single-node architecture, which is the most common computer architecture within the PAM community.

As a reference, we also provide in figure 3.3 the execution times of each version in a single node / single CPU mode, which corresponds to the minimal computer resource we can set.

3.1.2 Sensitivity studies of systems

Figure 3.4 and 3.5 compare computational performance of each system using different number of threads, written after the underscore in the system names (e.g. “MatlabSys_2” uses two threads).

3.2 Multiple node experimentation

Figures 3.6 and 3.7 represent the speed up metric of SparkSysOpti relatively to MatlabSys, as a function of the number of nodes for different workloads (in GB) for parameter sets 1 and 2, respectively. These results show that above 200 GB, an almost linear increase in speed is achieved. Indeed, as the workload increases, speed up linearizes towards the ideal case of scalability represented by the dashed black curve. For example, for parameter set 1 with a 33 GB workload, execution time only decreases by 3 when going from 1 to 4 cluster nodes, and further increasing the number of nodes up to 16 nodes does not decrease this time correspondingly. On the contrary, with a 300 GB workload, a decrease of execution time by almost 12 is observed over the increase of cluster nodes from 1 to 16. As this result has been obtained without specific optimization process, e.g. adapting the number of executors to the split length of audio file, it is very promising for further development towards a more general-purpose cloud-based analytics engine. It is noteworthy here that high-level frameworks like Hadoop / Spark highly facilitate access and democratize the use of distributed computing. On the contrary, frameworks like MPI (Message Passing Interface) would likely need more complex hand-code and fine tuning (e.g. manually setting chunking size, worker task and their synchronization), and require programming skills that are often well beyond competences of most computational scientists and researchers Dunner et al. (2017).

As expected, our proposed system SparkSysOpti does not perform well for small-volume datasets (approximately below 250 GB in our case), as a lot of executors are made available for a small number of tasks, resulting in a lot of unused resources. One way to boost scalability here would be simply to reduce the granularity of computations, i.e. reduce the Hadoop block size so that more tasks are created and more executors work simultaneously. Similarly, running executors with too much memory often results in excessive garbage collection delays, while running tiny executors (with a single core and just enough memory needed to run a single task, for example) throws away the benefits that come from running multiple in a single JVM.

Also, the different scalability behaviours observed across the different parameter sets can easily be explained. In set 1, the system scalability is stronger than set 2 as the number of operations to be performed (i.e. FFT computations and integration over segmentation windows) per executor for each wav is more important, as we can see at the last line of table 2.1, resulting in a smaller IO response time for the system (i.e. smaller waiting times for the workers).

Chapter 4 General discussion

Overall, in addition to this capacity of leveraging complex analytics, we believe that Hadoop and Spark should help to reshape the big data landscape in the field of PAM research for at least three other reasons. First, Spark is able to capture fairly general computations and facilitates the implementation of iterative algorithms, e.g. used for the training algorithms of machine learning systems111Spark’s machine learning library MLlib, made interoperable with NumPy, which now play an important role in most PAM applications (e.g. for whale detection and classification, see the DCLDE workshops). It also facilitates the implementation of interactive/exploratory data analysis (i.e., the repeated database-style querying of data), especially through its SQL-compliant query capability allowing user-defined functions that leverage any general-purpose function to apply to the data columns (e.g. to rank or aggregate rows of data over a sliding window). Such computational functionalities, made here at scale with speed, are now crucial in the context of big ocean data where PAM metrics are processed conjointly with multiple heterogeneous time series from other sensors. As a result, although we focus in this work on simple FFT-based descriptor computations, we envision our Apache Hadoop/Spark big data ecosystem growing as a general-purpose analysis system useful for many different types of PAM analysis. Third, numerous efforts have been made so far to outline some best practices for PAM processing (Robinson et al., 2014; Merchant et al., 2015), in the hope of boosting standardization and interoperability. On the contrary to expensive proprietary softwares, we believe that open source software like Apache Spark will strongly contribute to this dynamic, and we would encourage computational scientists and researchers to leave behind them “academic” codes that are too often made unreproducible, unbuildable, undocumented, unmaintained and backward incompatible.