Astrophysics and Big Data: Challenges, Methods, and Tools

TL;DR

This paper discusses the challenges posed by the increasing volume, variety, and velocity of astrophysical data, and explores machine learning methods and architectures for effective big data analysis in astrophysics.

Contribution

It reviews current challenges and proposes solutions involving machine learning and big data tools tailored for astrophysical data analysis.

Findings

Machine learning models can improve data processing efficiency.

Advanced architectures are essential for handling petabyte-scale data.

New tools are needed for complex, high-velocity astrophysical data.

Abstract

Nowadays there is no field research which is not flooded with data. Among the sciences, Astrophysics has always been driven by the analysis of massive amounts of data. The development of new and more sophisticated observation facilities, both ground-based and spaceborne, has led data more and more complex (Variety), an exponential growth of both data Volume (i.e., in the order of petabytes), and Velocity in terms of production and transmission. Therefore, new and advanced processing solutions will be needed to process this huge amount of data. We investigate some of these solutions, based on machine learning models as well as tools and architectures for Big Data analysis that can be exploited in the astrophysical context.

Tables

Table 1. Table 1: Big Data 3V characteristics in astronomical sky surveys.

Sky Survey		Volume	Velocity	Variety
SDSS Sloan Digital Sky Survey		$50$ TB	$200$ GB per day	images, catalogs, redshits
GAIA		$100$ TB	$40$ GB per day	more then $100$ parameters
Pan-STARRS Panoramic Survey Telescope and Rapid Response System		$5$ PB	$5$ TB per day	images, catalogs
LSST Large Synoptic Survey Telescope		$60$ PB	$10$ TB per day	images, catalogs
SKA Square Kilometer Array		$3$ ZB	$150$ TB per day	images, catalog, redshifts

Table 2. Table 2: Machine Learning Algorithms for Astrophysics

		Astrophysics Application		Machine Learning Algorithms
Supervised
Classification		AGN Classification, Globular Cluster Classification, Photometric Classification		Neural Network, Genetic Algorithms, Random Forest, Support Vector Machine, Bayesian Network, K-Nearest Neighbors
Regression		Photometric redshifts estimation		Linear Regression, Random Forest, Neural Network, K-Nearest Neighbors
Unsupervised
Clustering		Outlier Detection		Principal Component Analysis, K-Means, Self-organizing Map
Dimensional Reduction		Parameter space reduction		Principal Component Analysis, Probabilistic Principal Surface

Full text

††volume: 325††journal: Astroinformatics (AstroInfo16)††editors: A.C. Editor, B.D. Editor & C.E. Editor, eds.

1 Introduction

In the last decades, the exponential growth of data has changed the way we do science. New and increasingly sophisticated astronomical facilities, both ground-based and spaceborne, produce massive amounts of data and they will be able to reach in few years a production rate in the order of Petabyte/Year. This data tsunami, both in terms of volume and velocity, will bring Astronomy in the big data era. In many situations where it could be infeasible to store all data produced by facilities, it is crucial to efficiently analyze these data and to produce a response close to real time. So, machine learning (ML) algorithms, combined with big data analytics (BDA) architectures, become necessary tools in knowledge data discovery process, helping to automatically recognize hidden patterns inside the data and to understand their relationships. In this paper, we provide an overview of BDA in Astrophysics describing the big data generated by the sky surveys, the ML algorithms and the BDA platforms that could help to gain more meaningful insight on these datasets.

2 Big Data 3V in Astrophysics

What is Big Data? According to [Manyika (2011), Manyika et al. 2011] definition “Big data is datasets whose size is beyond the ability of typical database software tools to capture, store, manage, and analyze”. This definition contains a time-variant aspect. Datasets that today can be considered as Big Data tomorrow could become “normal” data. Since this definition does not use any metric to define big data, we prefer to use [Laney (2001), Laney et al. 2001] definition that identifies the data growth challenge as three-dimensional, i.e., concerning an increase in Volume, Velocity, and Variety.

Following the latter definition, Table 1 shows modern sky surveys in terms of the 3V characteristics.

• •

Volume refers to the amount of data. Different surveys produce datasets measurable in terabytes, petabytes, and even exabytes. This abundance forces to face challenges to capture, clean, transfer, store, analyze and visualize datasets.

• •

Velocity refers to both the data generation rate and the processing time requirement. Depending on the speed of data arrival, they can be processed in batch, if they arrive at intervals of time, or streaming if they require a real-time analysis (e.g. during the cleaning phase). Table 1 depicts surveys where the generation data is anticipated to exceed terabytes each night rate.

• •

Variety refers to the data type, i.e., structured, semi-structured, unstructured, and mixed. Astronomical sky surveys could include images, redshifts, time series data, and simulation data. Data from various sources have their formats, which causes the challenge of integrating data, and they have hundreds of features, introducing the problem of high dimensional data visualization.

3 Machine Learning Methods

Machine learning was defined in 1959 by Arthur Samuel as “the field of study that gives computers the ability to learn without being explicitly programmed”. ML allows to uncover hidden correlation patterns through an iterative learning by sample data (or past experiences) instead of being explicitly programmed. Common classes of problems that ML algorithms can solve are classification, regression, clustering, and outlier detection. These algorithms have been successfully used in astrophysics to solve different tasks. [DÍsanto et al. (2016), D’Istanto et al. 2016], presented an extensive investigation about classification performance of Random Forests, Multi-Layer Perceptron (MLP) with Quasi-Newton Algorithm, and K-Nearest Neighbors to classify transient objects, through experiments on the identification of cataclysmic variables, the separation between galactic and extra-galactic objects and identification of supernovae. [Masters et al. (2015), Masters et al. 2015] applied the Self-Organizing Maps (SOM) to the photometric redshifts problem, mapping the empirical distribution of galaxies in a multidimensional color space. A review of the use of data mining in astronomy was presented by [Tagliaferri et al. (2003), Tagliaferri et al. (2003)]. Table 2 gives an overview on ML methods used to face some astrophysical problems.

4 A data mining tool for astrophysics: DAMEWARE

DAMEWARE (Data Mining and Exploration Web Application REsource) is a web-based distributed platform for data mining with machine learning methods ([Brescia et al. (2014), Brescia et al. 2014]. The current release offers: as supervised methods four MLP implementations (i.e. classic Back Propagation, Genetic Algorithm, Quasi-Newton, and Levenberg-Marquardt Optimization Network), Random Forests, and Support Vector Machines (SVM); six SOM implementations, Principal Probabilistic Surfaces (PPS) and K-Means as unsupervised models. This set of techniques is useful for classification, regression, clustering, and feature selection tasks. DAMEWARE aims at to be an invaluable machine learning toolkit to face astronomical tasks such as AGN classification, photometric redshifts prediction, and globular cluster classification.

5 Big Data Analytics Platforms

BDA architectures, deployed on Cloud or In-House Data Center, have become critical to face the computationally demand ML algorithms. [Apache Hadoop, Apache Hadoop] is an open source platform for distributed storage and batch processing of large data sets on clusters built from commodity hardware. Hadoop is widely adopted by many leading institutions for educational or production uses. Its services provide data access (HIVE and HBase), job scheduling (YARN), a distributed file system (HDFS), and data processing (Map-Reduce). [Apache Spark, Apache Spark] is an open source BDA framework which makes it possible to analyze tons of data both in batch and streaming way. It provides two libraries for implementing machine learning: ML Lib, which provides ML models, and ML Pipelines handling the ML workflow, (i.e. data preparation, post-processing, and validations), helping to prepare and deploy the aforementioned models in a production environment. For both frameworks, Fig.  1 shows a schematic representation of the architecture and the steps involved in the analytics process. Several vendors, such as Amazon, Google, and Microsoft, make available these frameworks as cloud services. [Amazon EMR, Amazon Elastic MapReduce] (EMR) provides an Hadoop cluster distributing the computation across multiple [Amazon EC2, Amazon EC2] instances. Moreover, EMR can run processing frameworks, such as Apache Spark and HBase, and interact with data in other [Amazon AWS, Amazon Web Services] (AWS) data stores. Google, with [Cloud Dataproc, Cloud Dataproc], provides Apache Spark services on Hadoop clusters for batch processing, querying, streaming, and machine learning. Finally, Microsoft, with [Azure HDInsight, Azure HDInsight] provides an HDP Hadoop distribution in the cloud.

6 Conclusions

The aim of this paper was to provide a brief overview of big data challenges in astrophysics. Astrophysical use cases which successfully exploit machine learning algorithms were presented. Some cloud implementations of BDA platforms have been cited as systems able to efficiently and quickly execute these computationally demanding algorithms. They will be interesting to face with astrophysics big data challenges, but currently, data transfer technologies are unsuitable for the amount of data involved in the computing. Thus, we keep following the “move computing to the data” paradigm.