Cross-matching Engine for Incremental Photometric Sky Survey

TL;DR

This paper presents a novel approach to generate light curves from existing astronomical data without pre-planned surveys, focusing on scalable clustering techniques for catalog creation.

Contribution

It introduces a scalable, performance-oriented clustering method for astronomical catalog generation that leverages relational and NoSQL databases, enabling data mining of existing survey data.

Findings

Evaluated multiple clustering solutions for performance and scalability.

Developed quality standards for catalog generation.

Compared relational databases with NoSQL and supercomputing approaches.

Abstract

For light curve generation, a pre-planned photometry survey is needed nowadays, where all of the exposure coordinates have to be given and don't change during the survey. This thesis shows it is not required and we can data-mine these light curves from astronomical data that was never meant for this purpose. With this approach, we can recycle all of the photometric surveys in the world and generate light curves of observed objects for them. This thesis is addressing mostly the catalog generation process, which is needed for creating the light curves. In practice, it focuses on one of the most important problems in astroinformatics which is clustering data volumes on Big Data scale where most of the traditional techniques stagger. We consider a wide variety of possible solutions from the view of performance, scalability, distributability, etc. We defined criteria for time and memory complexity which we evaluated for all of the tested solutions. Furthermore, we created quality standards which we also take into account when evaluating the results. We are using relational databases as a starting point of our implementation and compare them with the newest technologies potentially usable for solving our problem. These are noSQL Array databases or transferring the heavy computations of clustering towards supercomputers by using parallelism.