A New Plotly-Dash-based Query Infrastructure for the Keck Observatory Archive

TL;DR

This paper presents a new web-based query system for the Keck Observatory Archive using Plotly-Dash, R-tree indexing, and VO standards, enabling faster, more interactive data access and visualization.

Contribution

The work introduces a Python-based, VO-compliant query infrastructure with real-time data ingestion and interactive services, replacing the previous system at KOA.

Findings

Spatial searches are 20 times faster.

New services support rapid data discovery and visualization.

Beta services demonstrate improved user interaction.

Abstract

The Keck Observatory Archive (KOA) curates all observational data acquired at the W. M. Keck Observatory. The archive is expected to grow rapidly as complex new instruments are commissioned and as the expectations of archive users have expanded. In response, KOA has implemented a new Python-based, VO-compliant query infrastructure. This work is a continuation of the architectural design and technology selection identified at ADASS 2024. We have deployed real-time ingestion of newly acquired data and a dedicated interface for observers to manage these data. Our ADASS 2024 poster identified the new technologies chosen: Plotly-Dash, a low-code framework that exploits event-driven callbacks to simplify the handling of user interactions; R-tree spatial indexing to speed up spatial searches by x20; a VO-compliant TAP middleware, already in use at the NASA Exoplanet Archive and NEID archive; and mViewer, a visualization engine in the Montage Image Mosaic toolkit that is optimized for astronomy images. These technologies will underpin new services that can be hosted on web pages or in Jupyter notebooks, and when completed, will replace the current query infrastructure. We have completed two new services now in beta release. The first is the Data Discovery Service, a web-based dashboard that returns spatial and temporal queries of the entire archive in seconds. It supports filtering observations by keywords, previewing results in an interactive data grid, and visualizing images, and it offers data downloads. The second is a Jupyter notebook that performs interactive visualization of Keck observations of protostars in the Rho Oph Dark Cloud and uses data from CDS and IRSA, as well as KOA.