Detection of Signals in Colored Noise: Roy's Largest Root Test for Non-central $F$-matrices

TL;DR

This paper develops a statistical framework for detecting signals in colored noise using Roy's largest root test, deriving the distribution of the leading eigenvalue of a non-central F-matrix, and analyzing its performance in high-dimensional settings.

Contribution

It provides a closed-form distribution of the largest eigenvalue of a non-central F-matrix and extends the analysis to high-dimensional regimes for signal detection.

Findings

The derived c.d.f. enables ROC analysis of the detector.

Detection performance improves with higher SNR, especially when SNR is at least O(p^2) in high dimensions.

Weak signals are undetectable in high dimensions when m<n with SNR of order O(p).

Abstract

This paper investigates the signal detection problem in colored noise with an unknown covariance matrix. In particular, we focus on detecting a non-random signal by capitalizing on the leading eigenvalue (a.k.a. Roy's largest root) of the whitened sample covariance matrix as the test statistic. To this end, the whitened sample covariance matrix is constructed via \(m\)-dimensional \(p \) plausible signal-bearing samples and \(m\)-dimensional \(n \) noise-only samples. Since the signal is non-random, the whitened sample covariance matrix turns out to have a {\it non-central} \(F\)-distribution with a rank-one non-centrality parameter. Therefore, the performance of the test entails the statistical characterization of the leading eigenvalue of the non-central \(F\)-matrix, which we address by deriving its cumulative distribution function (c.d.f.) in closed-form by leveraging the powerful orthogonal polynomial approach in random matrix theory. This new c.d.f. has been instrumental in analyzing the receiver operating characteristic (ROC) of the detector. We also extend our analysis into the high dimensional regime in which \(m,n\), and \(p\) diverge such that \(m/n\) and \(m/p\) remain fixed. It turns out that, when \(m=n\) and fixed, the power of the test improves if the signal-to-noise ratio (SNR) is of at least \(O(p)\), whereas the corresponding SNR in the high dimensional regime is of at least \(O(p^2)\). Nevertheless, more intriguingly, for \(m<n\) with the SNR of order \(O(p)\), the leading eigenvalue does not have power to detect {\it weak} signals in the high dimensional regime.