TokenCom: Vision-Language Model for Multimodal and Multitask Token Communications

TL;DR

This paper introduces TaiChi, a novel vision-language model framework that enhances token communication by capturing multi-scale visual features and achieving precise cross-modal alignment, improving multimodal understanding and task performance.

Contribution

The paper presents TaiChi, a new VLM framework with dual-visual tokenization, Bilateral Attention Network, and KAN-based modality projection for improved multimodal token communication.

Findings

TaiChi outperforms existing models in visual understanding tasks.

The system demonstrates effective multimodal and multitask token communication.

Experimental results confirm the model's superior performance and feasibility.

Abstract

Visual-Language Models (VLMs), with their strong capabilities in image and text understanding, offer a solid foundation for intelligent communications. However, their effectiveness is constrained by limited token granularity, overlong visual token sequences, and inadequate cross-modal alignment. To overcome these challenges, we propose TaiChi, a novel VLM framework designed for token communications. TaiChi adopts a dual-visual tokenizer architecture that processes both high- and low-resolution images to collaboratively capture pixel-level details and global conceptual features. A Bilateral Attention Network (BAN) is introduced to intelligently fuse multi-scale visual tokens, thereby enhancing visual understanding and producing compact visual tokens. In addition, a Kolmogorov Arnold Network (KAN)-based modality projector with learnable activation functions is employed to achieve precise nonlinear alignment from visual features to the text semantic space, thus minimizing information loss. Finally, TaiChi is integrated into a multimodal and multitask token communication system equipped with a joint VLM-channel coding scheme. Experimental results validate the superior performance of TaiChi, as well as the feasibility and effectiveness of the TaiChi-driven token communication system.