Coverage Analysis of Multi-Environment Q-Learning Algorithms for Wireless Network Optimization

TL;DR

This paper analyzes coverage conditions for multi-environment Q-learning algorithms in wireless network optimization, proposing an initialization method that improves accuracy and reduces complexity, validated through real-world tests.

Contribution

It introduces a coverage analysis framework and an initialization algorithm for ensemble multi-environment Q-learning, enhancing performance and robustness in wireless networks.

Findings

Achieves 50% less policy error

Reduces runtime complexity by 40%

Demonstrates robustness to network changes

Abstract

Q-learning is widely used to optimize wireless networks with unknown system dynamics. Recent advancements include ensemble multi-environment hybrid Q-learning algorithms, which utilize multiple Q-learning algorithms across structurally related but distinct Markovian environments and outperform existing Q-learning algorithms in terms of accuracy and complexity in large-scale wireless networks. We herein conduct a comprehensive coverage analysis to ensure optimal data coverage conditions for these algorithms. Initially, we establish upper bounds on the expectation and variance of different coverage coefficients. Leveraging these bounds, we present an algorithm for efficient initialization of these algorithms. We test our algorithm on two distinct real-world wireless networks. Numerical simulations show that our algorithm can achieve %50 less policy error and %40 less runtime complexity than state-of-the-art reinforcement learning algorithms. Furthermore, our algorithm exhibits robustness to changes in network settings and parameters. We also numerically validate our theoretical results.