Robust Machine Learning Applied to Terascale Astronomical Datasets

TL;DR

This paper demonstrates the application of advanced machine learning techniques and supercomputing resources to improve classifications and redshift estimations for over 100 million objects in the SDSS, marking a significant step in handling terascale astronomical data.

Contribution

It introduces the first comprehensive application of decision trees, k-nearest neighbor, and genetic algorithms to terascale astronomical datasets, enhancing data analysis capabilities.

Findings

Improved classifications for over 100 million SDSS objects.

Enhanced photometric redshift accuracy.

Demonstrated scalability of algorithms to terascale data.

Abstract

We present recent results from the Laboratory for Cosmological Data Mining (http://lcdm.astro.uiuc.edu) at the National Center for Supercomputing Applications (NCSA) to provide robust classifications and photometric redshifts for objects in the terascale-class Sloan Digital Sky Survey (SDSS). Through a combination of machine learning in the form of decision trees, k-nearest neighbor, and genetic algorithms, the use of supercomputing resources at NCSA, and the cyberenvironment Data-to-Knowledge, we are able to provide improved classifications for over 100 million objects in the SDSS, improved photometric redshifts, and a full exploitation of the powerful k-nearest neighbor algorithm. This work is the first to apply the full power of these algorithms to contemporary terascale astronomical datasets, and the improvement over existing results is demonstrable. We discuss issues that we have encountered in dealing with data on the terascale, and possible solutions that can be implemented to deal with upcoming petascale datasets.