Generative Semantic Communication via Textual Prompts: Latency Performance Tradeoffs

TL;DR

This paper introduces a collaborative semantic communication framework using pre-trained multi-modal models to optimize visual prompt generation and transmission, achieving low latency and high semantic fidelity in wireless edge devices.

Contribution

It develops a novel multi-user generative semantic communication framework with joint optimization of prompt offloading, resource allocation, and latency reduction.

Findings

Significant latency reduction compared to benchmarks.

Enhanced semantic quality through optimized prompt generation.

Effective multi-user collaboration with low complexity algorithms.

Abstract

This paper develops an edge-device collaborative Generative Semantic Communications (Gen SemCom) framework leveraging pre-trained Multi-modal/Vision Language Models (M/VLMs) for ultra-low-rate semantic communication via textual prompts. The proposed framework optimizes the use of M/VLMs on the wireless edge/device to generate high-fidelity textual prompts through visual captioning/question answering, which are then transmitted over a wireless channel for SemCom. Specifically, we develop a multi-user Gen SemCom framework using pre-trained M/VLMs, and formulate a joint optimization problem of prompt generation offloading, communication and computation resource allocation to minimize the latency and maximize the resulting semantic quality. Due to the nonconvex nature of the problem with highly coupled discrete and continuous variables, we decompose it as a two-level problem and propose a low-complexity swap/leaving/joining (SLJ)-based matching algorithm. Simulation results demonstrate significant performance improvements over the conventional semanticunaware/non-collaborative offloading benchmarks.