ReaCritic: Reasoning Transformer-based DRL Critic-model Scaling For Wireless Networks

TL;DR

ReaCritic introduces a transformer-based critic model that incorporates reasoning capabilities into DRL, significantly improving convergence speed and performance in wireless network management and control tasks.

Contribution

It presents ReaCritic, a novel reasoning transformer-based critic model that enhances DRL adaptability and generalization in complex, dynamic wireless environments.

Findings

Improves convergence speed in HetNet scenarios

Enhances final performance across various tasks

Compatible with multiple DRL algorithms

Abstract

Heterogeneous Networks (HetNets) pose critical challenges for intelligent management due to the diverse user requirements and time-varying wireless conditions. These factors introduce significant decision complexity, which limits the adaptability of existing Deep Reinforcement Learning (DRL) methods. In many DRL algorithms, especially those involving value-based or actor-critic structures, the critic component plays a key role in guiding policy learning by estimating value functions. However, conventional critic models often use shallow architectures that map observations directly to scalar estimates, limiting their ability to handle multi-task complexity. In contrast, recent progress in inference-time scaling of Large Language Models (LLMs) has shown that generating intermediate reasoning steps can significantly improve decision quality. Motivated by this, we propose ReaCritic, a reasoning transformer-based critic-model scaling scheme that brings reasoning-like ability into DRL. ReaCritic performs horizontal reasoning over parallel state-action inputs and vertical reasoning through deep transformer stacks. It is compatible with a broad range of value-based and actor-critic DRL algorithms and enhances generalization in dynamic wireless environments. Extensive experiments demonstrate that ReaCritic improves convergence speed and final performance across various HetNet settings and standard OpenAI Gym control tasks. The code of ReaCritic is available at https://github.com/NICE-HKU/ReaCritic.