# Eigenvalue Based Detection of a Signal in Colored Noise: Finite and Asymptotic Analyses

## TL;DR

This paper derives the finite-sample and asymptotic distribution of the largest generalized eigenvalue used for signal detection in colored noise, enabling improved ROC analysis and detection performance understanding.

## Contribution

It provides the first finite-dimensional characterization of the eigenvalue distribution under the alternative hypothesis using orthogonal polynomial methods.

## Key findings

- Finite-sample c.d.f. of the largest generalized eigenvalue derived.
- Asymptotic c.d.f. and special cases analyzed.
- Reliable detection possible when SNR scales with sample size.

## Abstract

Signal detection in colored noise with an unknown covariance matrix has a myriad of applications in diverse scientific/engineering fields. The test statistic is the largest generalized eigenvalue (l.g.e.) of the whitened sample covariance matrix, which is constructed via $m$-dimensional $p $ signal-plus-noise samples and $m$-dimensional $n $ noise-only samples. A finite dimensional characterization of this statistic under the alternative hypothesis has hitherto been an open problem. We answer this problem by deriving cumulative distribution function (c.d.f.) of this l.g.e. via the powerful orthogonal polynomial approach, exploiting the deformed Jacobi unitary ensemble (JUE). Two special cases and an asymptotic version of the c.d.f. are also derived. With this new c.d.f., we comprehensively analyze the receiver operating characteristics (ROC) of the detector. Importantly, when the noise-only covariant matrix is nearly rank deficient (i.e., $ m=n$), we show that (a) when $m$ and $p$ increase such that $m/p$ is fixed, at each fixed signal-to-noise ratio (SNR), there exists an optimal ROC profile. We also establish a tight approximation of it; and (b) asymptotically, reliable signal detection is always possible (no matter how weak the signal is) if SNR scales with $m$.