Integration of Data Reduction and Near Real-Time Archiving into the Keck Observing Model

TL;DR

The Keck Observatory is integrating data reduction and near real-time archiving into its observing model, enabling immediate access to raw and processed data for observers and the public, enhancing collaboration and data sharing.

Contribution

This paper introduces a new integrated data reduction and archiving system with near real-time capabilities at Keck Observatory.

Findings

Implementation of near real-time data archiving system

Immediate data access for observers and collaborators

Public availability of raw and processed data after protections

Abstract

The W. M. Keck Observatory is welcoming a new era where data reduction and archiving are tightly integrated into our observing model, under the auspices of the Observatory's Data Services Initiative (DSI) project. While previously the Keck Observatory Archive (KOA) archived minimally processed, raw science data the day after observing, Keck is transitioning to a model in which it archives both raw frames and reduced data in near real-time. These data will be made available to observers and collaborators immediately upon ingestion through a dedicated new interface that will support collaboration and sharing among teams, as well as stream data directly to personal computers without access to WMKO's internal networks. Both the raw and science-ready data products will be made publicly available upon the expiration of data protections.