Causal Model-Based Reinforcement Learning for Sample-Efficient IoT Channel Access

TL;DR

This paper introduces a causal model-based reinforcement learning framework for IoT channel access that improves sample efficiency and interpretability by explicitly modeling causal relationships, enabling faster learning and better decision explanations.

Contribution

The paper develops a novel causal model-based MARL framework using structural causal models and attention mechanisms, enhancing interpretability and sample efficiency over traditional black-box methods.

Findings

Achieves 58% reduction in environment interactions.

Demonstrates exponential sample complexity gains.

Provides interpretable scheduling decisions via causal attribution.

Abstract

Despite the advantages of multi-agent reinforcement learning (MARL) for wireless use case such as medium access control (MAC), their real-world deployment in Internet of Things (IoT) is hindered by their sample inefficiency. To alleviate this challenge, one can leverage model-based reinforcement learning (MBRL) solutions, however, conventional MBRL approaches rely on black-box models that are not interpretable and cannot reason. In contrast, in this paper, a novel causal model-based MARL framework is developed by leveraging tools from causal learn- ing. In particular, the proposed model can explicitly represent causal dependencies between network variables using structural causal models (SCMs) and attention-based inference networks. Interpretable causal models are then developed to capture how MAC control messages influence observations, how transmission actions determine outcomes, and how channel observations affect rewards. Data augmentation techniques are then used to generate synthetic rollouts using the learned causal model for policy optimization via proximal policy optimization (PPO). Analytical results demonstrate exponential sample complexity gains of causal MBRL over black-box approaches. Extensive simulations demonstrate that, on average, the proposed approach can reduce environment interactions by 58%, and yield faster convergence compared to model-free baselines. The proposed approach inherently is also shown to provide interpretable scheduling decisions via attention-based causal attribution, revealing which network conditions drive the policy. The resulting combination of sample efficiency and interpretability establishes causal MBRL as a practical approach for resource-constrained wireless systems.