BOOM and Babamul: a real-time, multi-survey, optical alert broker system operating at scale

TL;DR

The paper introduces BOOM, a high-throughput, real-time alert broker system for astronomical surveys, and discusses its development and capabilities for upcoming large-scale sky surveys like LSST.

Contribution

It presents a scalable, efficient, Rust-based framework for real-time processing of astronomical alerts, building on previous systems and preparing for LSST's data volume.

Findings

BOOM achieves 7 times higher throughput than previous ZTF systems.

The system maintains feature parity with existing alert brokers.

Demonstrated real-time processing with custom filters.

Abstract

With the arrival of ever higher throughput wide-field surveys and a multitude of multi-messenger and multi-wavelength instruments to complement them, software capable of harnessing these associated data streams is urgently required. To meet these needs, a number of community supported alert brokers have been built, currently focused on processing of Zwicky Transient Facility (ZTF; $\sim 10^5$-$10^6$ alerts per night) with an eye towards Vera C. Rubin Observatory's Legacy Survey of Space and Time (LSST; $\sim 2 \times 10^7$ alerts per night). Building upon the system that successfully ran in production for ZTF's first seven years of operation, we introduce BOOM (Burst & Outburst Observations Monitor), an analysis framework focused on real-time, joint brokering of these alert streams. BOOM harnesses the performance of a Rust-based software stack relying on a non-relational MongoDB database combined with a Valkey in-memory processing queue and a Kafka cluster for message sharing. With this system, we demonstrate feature parity with the existing ZTF system with a throughput $\sim 7 \times$ higher. We describe the workflow that enables the real-time processing as well as the results with custom filters we have built to demonstrate the system's capabilities. In conclusion, we present the development roadmap for both BOOM and Babamul - the public-facing LSST alert broker built atop BOOM - as we begin the Rubin era.