Reinforcement Learning for Robust Header Compression under Model Uncertainty

TL;DR

This paper introduces a reinforcement learning approach using deep Q-networks to enhance robust header compression in wireless systems, effectively handling model uncertainty and scalability issues.

Contribution

It presents a novel RL-based architecture for BD-ROHC that scales better than dynamic programming and does not require prior knowledge of environment dynamics.

Findings

RL approach outperforms traditional methods in complex scenarios

Scalable to large state and action spaces

Does not need prior environment model knowledge

Abstract

Robust header compression (ROHC), critically positioned between the network and the MAC layers, plays an important role in modern wireless communication systems for improving data efficiency. This work investigates bi-directional ROHC (BD-ROHC) integrated with a novel architecture of reinforcement learning (RL). We formulate a partially observable \emph{Markov} decision process (POMDP), in which agent is the compressor, and the environment consists of the decompressor, channel and header source. Our work adopts the well-known deep Q-network (DQN), which takes the history of actions and observations as inputs, and outputs the Q-values of corresponding actions. Compared with the ideal dynamic programming (DP) proposed in the existing works, our method is scalable to the state, action and observation spaces. In contrast, DP often suffers from formidable computational complexity when the number of states becomes large due to long decompressor feedback delay and complex channel models. In addition, our method does not require prior knowledge of the transition dynamics and accurate observation dependency of the model, which are often not available in many practical applications.