SMA-X: Versatile information sharing in and around telescopes, and beyond

TL;DR

SMA-X is a real-time, high-throughput data sharing system for telescopes, enabling efficient hierarchical data access and integration across observatory systems, and is openly available for community use.

Contribution

It introduces SMA-X, a versatile, low-latency data sharing platform built on Redis, with server-side scripting and tools for observatory data management and sharing.

Findings

Shared ~10,000 variables across 100+ computers since 2021

Supports real-time data updates every 10 ms

Open-source with multi-language libraries and tools

Abstract

We developed the SMA eXchange (SMA-X) as a real-time data sharing solution, built atop a central Redis database. SMA-X is a storage convention, facilitated by a set of server-side Lua scripts (or Redis functions) which enable efficient low-latency and high-throughput real-time sharing of hierarchically structured data among the various systems and subsystems of the telescope. It enables fast, atomic retrievals of specific leaf elements, branches, and sub-trees, including associated metadata (types, dimensions, timestamps, and origins, and more). At the Submillimer Array (SMA) we rely on it since 2021 to share a diverse set of ~10,000 real-time variables, including arrays, across more than 100 computers, with information being published every 10 ms in some cases. SMA-X is open-source, and is freely available to everyone through a set of public GitHub repositories, including C/C++ and Python3 libraries to allow integration with observatory applications. A set of command-line tools provide access to the database from the POSIX shell and/or from any scripting language, and we also provide a configurable system daemon for archiving the observatory state at regular intervals into a time-series Structured Query Language (SQL) database to create a detailed historical record.