Leveraging Data Lineage to Infer Logical Relationships between Astronomical Catalogs

TL;DR

This paper introduces a novel logical inference method for understanding relationships between astronomical catalogs using data lineage, enabling reasoning without explicit content knowledge, and demonstrates its application in the Astro-WISE system.

Contribution

The paper presents a new method for inferring logical relationships between data sets, specifically astronomical catalogs, using incomplete knowledge and data lineage, without needing explicit content.

Findings

Effective inference of catalog relationships demonstrated in Astro-WISE

Method handles large catalogs without accessing full contents

Logical reasoning improves data dependency understanding

Abstract

A novel method to infer logical relationships between sets is presented. These sets can be any collection of elements, for example astronomical catalogs of celestial objects. The method does not require the contents of the sets to be known explicitly. It combines incomplete knowledge about the relationships between sets to infer a priori unknown relationships. Relationships between sets are represented by sets of Boolean hypercubes. This leads to deductive reasoning by application of logical operators to these sets of hypercubes. A pseudocode for an efficient implementation is described. The method is used in the Astro-WISE information system to infer relationships between catalogs of astronomical objects. These catalogs can be very large and, more importantly, their contents do not have to be available at all times. Science products are stored in Astro-WISE with references to other science products from which they are derived, or their dependencies. This creates full data lineage that links every science product all the way back to the raw data. Catalogs are created in a way that maximizes knowledge about their relationship with their dependencies. The presented algorithm is used to determine which objects a catalog represents by leveraging this information.