Novelty Detection on Radio Astronomy Data using Signatures

TL;DR

SigNova is a semi-supervised framework that uses signature transforms and Mahalanobis distance to detect radio-frequency interference anomalies in streamed radio astronomy data, adaptable to various data types.

Contribution

The paper introduces SigNova, a novel semi-supervised anomaly detection method utilizing signature transforms and Mahalanobis distance, applicable beyond radio astronomy.

Findings

Effective detection of broadband and narrowband RFI.

Validated on MWA and HERA datasets.

Improves anomaly localization accuracy.

Abstract

We introduce SigNova, a new semi-supervised framework for detecting anomalies in streamed data. While our initial examples focus on detecting radio-frequency interference (RFI) in digitized signals within the field of radio astronomy, it is important to note that SigNova's applicability extends to any type of streamed data. The framework comprises three primary components. Firstly, we use the signature transform to extract a canonical collection of summary statistics from observational sequences. This allows us to represent variable-length visibility samples as finite-dimensional feature vectors. Secondly, each feature vector is assigned a novelty score, calculated as the Mahalanobis distance to its nearest neighbor in an RFI-free training set. By thresholding these scores we identify observation ranges that deviate from the expected behavior of RFI-free visibility samples without relying on stringent distributional assumptions. Thirdly, we integrate this anomaly detector with Pysegments, a segmentation algorithm, to localize consecutive observations contaminated with RFI, if any. This approach provides a compelling alternative to classical windowing techniques commonly used for RFI detection. Importantly, the complexity of our algorithm depends on the RFI pattern rather than on the size of the observation window. We demonstrate how SigNova improves the detection of various types of RFI (e.g., broadband and narrowband) in time-frequency visibility data. We validate our framework on the Murchison Widefield Array (MWA) telescope and simulated data and the Hydrogen Epoch of Reionization Array (HERA).