Context-Aware Wireless Token Communication via Joint Token Masking and Detection

TL;DR

This paper introduces a context-aware wireless token communication framework that uses a shared masked language model to improve transmission efficiency and robustness over noisy channels, outperforming traditional methods.

Contribution

It presents a novel joint token masking and detection approach leveraging MLM-based context modeling for efficient wireless token communication.

Findings

Achieves up to 1.77X performance gain on Europarl corpus.

Achieves up to 1.63X performance gain on WikiText-103.

Significantly outperforms conventional token communication schemes.

Abstract

The increasing use of token-based representations in language-driven applications has motivated wireless token communication, where tokens are treated as fundamental units for transmission. However, conventional communication systems overlook dependencies among tokens and allocate transmission resources uniformly, leading to inefficient use of limited wireless resources under channel impairments. In this paper, we propose a context-aware token communication framework that leverages a masked language model (MLM) as a shared contextual model between the transmitter (Tx) and receiver (Rx). At the Rx, we develop a context-aware token detection method that integrates channel likelihoods with MLM-based contextual priors under a Bayesian formulation, enabling robust token inference over noisy channels. At the Tx, we propose a context-aware token masking strategy that selectively omits tokens that can be reliably inferred at the Rx, allowing the available power budget to be concentrated on more informative tokens. These components are jointly designed through a shared MLM, establishing a unified Tx-Rx framework for efficient token transmission and detection. Simulation results demonstrate that the proposed framework significantly improves reconstruction performance compared to conventional and existing token communication schemes, achieving up to 1.77X and 1.63X performance gains on the Europarl corpus and WikiText-103 datasets, respectively.