Semantic Importance-Aware Communications with Semantic Correction Using Large Language Models

TL;DR

This paper introduces understanding-level semantic communications that convert visual data into natural language, using large language models for importance assessment and error correction, thereby improving semantic fidelity and privacy over traditional feature-level methods.

Contribution

It proposes a novel approach combining image captioning and large language models to enable semantic understanding and correction in communication systems, surpassing feature-level methods.

Findings

ULSC achieves higher semantic similarity than FLSC.

Semantic correction with LLM reduces semantic errors.

Enhances privacy by transmitting only natural language descriptions.

Abstract

Semantic communications, a promising approach for agent-human and agent-agent interactions, typically operate at a feature level, lacking true semantic understanding. This paper explores understanding-level semantic communications (ULSC), transforming visual data into human-intelligible semantic content. We employ an image caption neural network (ICNN) to derive semantic representations from visual data, expressed as natural language descriptions. These are further refined using a pre-trained large language model (LLM) for importance quantification and semantic error correction. The subsequent semantic importance-aware communications (SIAC) aim to minimize semantic loss while respecting transmission delay constraints, exemplified through adaptive modulation and coding strategies. At the receiving end, LLM-based semantic error correction is utilized. If visual data recreation is desired, a pre-trained generative artificial intelligence (AI) model can regenerate it using the corrected descriptions. We assess semantic similarities between transmitted and recovered content, demonstrating ULSC's superior ability to convey semantic understanding compared to feature-level semantic communications (FLSC). ULSC's conversion of visual data to natural language facilitates various cognitive tasks, leveraging human knowledge bases. Additionally, this method enhances privacy, as neither original data nor features are directly transmitted.