Task-Oriented Multimodal Token Transmission in Resource-Constrained Multiuser Networks

TL;DR

This paper introduces a task-oriented multimodal token transmission scheme for resource-constrained multiuser networks, reducing bandwidth and power consumption while maintaining high model performance through a novel training and optimization approach.

Contribution

It proposes a two-stage training algorithm with token compression and joint resource optimization for efficient multimodal token transmission in multiuser networks.

Findings

Outperforms baseline algorithms in bandwidth and power efficiency.

Achieves higher accuracy across various SNR levels.

Effectively balances latency and model performance.

Abstract

With the emergence of large model-based agents, widely adopted transformer-based architectures inevitably produce excessively long token embeddings for transmission, which may result in high bandwidth overhead, increased power consumption and latency. In this letter, we propose a task-oriented multimodal token transmission scheme for efficient multimodal information fusion and utilization. To improve the efficiency of token transmission, we design a two-stage training algotithm, including cross-modal alignment and task-oriented fine-tuning, for large model-based token communication. Meanwhile, token compression is performed using a sliding window pooling operation to save communication resources. To balance the trade-off between latency and model performance caused by compression, we formulate a weighted-sum optimization problem over latency and validation loss. We jointly optimizes bandwidth, power allocation, and token length across users by using an alternating optimization method. Simulation results demonstrate that the proposed algorithm outperforms the baseline under different bandwidth and power budgets. Moreover, the two-stage training algorithm achieves higher accuracy across various signal-to-noise ratios than the method without cross-modal alignment.