# Detection of a Signal in Colored Noise: A Random Matrix Theory Based Analysis

## TL;DR

This paper develops a random matrix theory-based method to detect signals in colored noise by analyzing the largest eigenvalue of a whitened covariance matrix, providing new statistical characterizations and optimal ROC profiles.

## Contribution

It introduces a finite-dimensional expression for the eigenvalue distribution of the deformed JUE, enabling improved detection analysis in colored noise environments.

## Key findings

- Derived a new c.d.f. for the largest eigenvalue of the deformed JUE.
- Established an optimal ROC profile for fixed SNR conditions.
- Provided tight approximations for detection performance in high-dimensional settings.

## Abstract

This paper investigates the classical statistical signal processing problem of detecting a signal in the presence of colored noise with an unknown covariance matrix. In particular, we consider a scenario where m-dimensional p possible signal-plus-noise samples and m-dimensional n noise-only samples are available at the detector. Then the presence of a signal can be detected using the largest generalized eigenvalue (l.g.e.) of the so called whitened sample covariance matrix. This amounts to statistically characterizing the maximum eigenvalue of the deformed Jacobi unitary ensemble (JUE). To this end, we employ the powerful orthogonal polynomial approach to determine a new finite dimensional expression for the cumulative distribution function (c.d.f.) of the l.g.e. of the deformed JUE. This new c.d.f. expression facilitates the further analysis of the receiver operating characteristics (ROC) of the detector. It turns out that, for m=n, when m and p increase such that m/p attains a fixed value, there exists an optimal ROC profile corresponding to each fixed signal-to-noise ratio (SNR). In this respect, we have established a tight approximation for the corresponding optimal ROC profile.