RL-Aided Cognitive ISAC: Robust Detection and Sensing-Communication Trade-offs

TL;DR

This paper introduces an RL-based cognitive framework for massive MIMO ISAC systems that adaptively balances radar detection accuracy and communication throughput, improving robustness in dynamic environments.

Contribution

It presents a novel RL-aided approach for adaptive sensing and waveform optimization in massive MIMO ISAC systems, addressing unknown disturbance characteristics.

Findings

Enhanced detection probability over baseline methods.

Achieved adaptive trade-off between sensing and communication performance.

Demonstrated robustness in dynamic, non-Gaussian environments.

Abstract

This paper proposes a reinforcement learning (RL)-aided cognitive framework for massive MIMO-based integrated sensing and communication (ISAC) systems employing a uniform planar array (UPA). The focus is on enhancing radar sensing performance in environments with unknown and dynamic disturbance characteristics. A Wald-type detector is employed for robust target detection under non-Gaussian clutter, while a SARSA-based RL algorithm enables adaptive estimation of target positions without prior environmental knowledge. Based on the RL-derived sensing information, a joint waveform optimization strategy is formulated to balance radar sensing accuracy and downlink communication throughput. The resulting design provides an adaptive trade-off between detection performance and achievable sum rate through an analytically derived closed-form solution. Monte Carlo simulations demonstrate that the proposed cognitive ISAC framework achieves significantly improved detection probability compared to orthogonal and non-learning adaptive baselines, while maintaining competitive communication performance. These results underline the potential of RL-assisted sensing for robust and spectrum-efficient ISAC in next-generation wireless networks.