Out-of-Distribution Radar Detection with Complex VAEs: Theory, Whitening, and ANMF Fusion

TL;DR

This paper introduces a complex-valued variational autoencoder (CVAE) for detecting weak signals in maritime radar clutter, demonstrating improved detection performance over classical methods through simulation and real data analysis.

Contribution

It proposes a novel CVAE-based out-of-distribution detector for complex radar signals, with whitening and fusion techniques to enhance robustness and detection accuracy.

Findings

CVAE outperforms classical detectors in detection probability.

Whitening significantly improves detection performance.

Fusion of CVAE with ANMF enhances robustness in non-Gaussian clutter.

Abstract

We investigate the detection of weak complex-valued signals immersed in non-Gaussian, range-varying interference, with emphasis on maritime radar scenarios. The proposed methodology exploits a Complex-valued Variational AutoEncoder (CVAE) trained exclusively on clutter-plus-noise to perform Out-Of-Distribution detection. By operating directly on in-phase / quadrature samples, the CVAE preserves phase and Doppler structure and is assessed in two configurations: (i) using unprocessed range profiles and (ii) after local whitening, where per-range covariance estimates are obtained from neighboring profiles. Using extensive simulations together with real sea-clutter data from the CSIR maritime dataset, we benchmark performance against classical and adaptive detectors (MF, NMF, AMF-SCM, ANMF-SCM, ANMF-Tyler). In both configurations, the CVAE yields a higher detection probability Pd at matched false-alarm rate Pfa, with the most notable improvements observed under whitening. We further integrate the CVAE with the ANMF through a weighted log-p fusion rule at the decision level, attaining enhanced robustness in strongly non-Gaussian clutter and enabling empirically calibrated Pfa control under H0. Overall, the results demonstrate that statistical normalization combined with complex-valued generative modeling substantively improves detection in realistic sea-clutter conditions, and that the fused CVAE-ANMF scheme constitutes a competitive alternative to established model-based detectors.