New Python-based Architecture for the Keck Observatory Archive

TL;DR

This paper introduces a Python-based, VO-compliant data discovery service for the Keck Observatory Archive, enabling rapid, multi-instrument data queries and visualization through a modern web interface.

Contribution

It presents a new Python infrastructure using nexsciTAP and R-tree spatial indexing for fast, scalable archive searches, integrated with a user-friendly web dashboard.

Findings

Enables 2-second searches of 100 million records

Speeds up spatial searches by 20 times

Supports exploration and visualization across multiple instruments

Abstract

We describe the development of the Keck Observatory Archive (KOA) Data Discovery Service, a web-based dashboard that returns metadata for wide-area queries of the entire archive in seconds. Currently in beta, this dashboard will support exploration, visualization, and data access across multiple instruments. This effort is underpinned by an open-source, VO-compliant query infrastructure and will offer services that can be hosted on web pages or in Jupyter notebooks. The effort also informs the design of a new, modern landing page that meets the expectations of accessibility and ease of use. The new query infrastructure is based on nexsciTAP, a component-based, DBMS-agnostic Python implementation of the IVOA Table Access Protocol, developed at NExScI and integrated into the NASA Exoplanet Archive and the NEID Archive, and into the PyKOA Python client. This infrastructure incorporates R-tree spatial indexing, built as memory-mapped files as part of Montage, a software toolkit used to create composite astronomical images. Although R-trees are used most often in geospatial analysis, here they enable searches of the entire KOA archive, an eclectic collection of 100 million records of imaging and spectroscopic data, in 2 seconds, and they speed up spatial searches by x20. The front end is built on the open-source Plotly-Dash framework, which allows users to build an interactive user interface based on a single Python file.