Fundamental limit of sample generalized eigenvalue based detection of signals in noise using relatively few signal-bearing and noise-only samples

TL;DR

This paper establishes the fundamental asymptotic limit for detecting signals in noise using sample generalized eigenvalues, especially when the number of samples is relatively small, impacting various fields like radar, wireless communications, and machine learning.

Contribution

It proves a fundamental limit for signal detection via generalized eigenvalues in colored noise with few samples, extending understanding of detection capabilities in practical scenarios.

Findings

Analytical prediction matches numerical simulations

Defines the effective number of identifiable signals in colored noise

Implications for detecting weak and closely spaced signals

Abstract

The detection problem in statistical signal processing can be succinctly formulated: Given m (possibly) signal bearing, n-dimensional signal-plus-noise snapshot vectors (samples) and N statistically independent n-dimensional noise-only snapshot vectors, can one reliably infer the presence of a signal? This problem arises in the context of applications as diverse as radar, sonar, wireless communications, bioinformatics, and machine learning and is the critical first step in the subsequent signal parameter estimation phase. The signal detection problem can be naturally posed in terms of the sample generalized eigenvalues. The sample generalized eigenvalues correspond to the eigenvalues of the matrix formed by "whitening" the signal-plus-noise sample covariance matrix with the noise-only sample covariance matrix. In this article we prove a fundamental asymptotic limit of sample generalized eigenvalue based detection of signals in arbitrarily colored noise when there are relatively few signal bearing and noise-only samples. Numerical simulations highlight the accuracy of our analytical prediction and permit us to extend our heuristic definition of the effective number of identifiable signals in colored noise. We discuss implications of our result for the detection of weak and/or closely spaced signals in sensor array processing, abrupt change detection in sensor networks, and clustering methodologies in machine learning.