Large Generative Model Assisted 3D Semantic Communication

TL;DR

This paper introduces GAM-3DSC, a system leveraging generative AI for efficient 3D semantic communication in 6G, addressing semantic extraction, redundancy, and channel estimation challenges with novel models and methods.

Contribution

The paper proposes a comprehensive framework combining generative AI models, semantic compression, and channel estimation techniques specifically for 3D semantic communication in 6G networks.

Findings

Effective 3D semantic extraction using SAM and NeRF.

Redundancy masking improves compression efficiency.

Simulation shows enhanced transmission of 3D scenarios.

Abstract

Semantic Communication (SC) is a novel paradigm for data transmission in 6G. However, there are several challenges posed when performing SC in 3D scenarios: 1) 3D semantic extraction; 2) Latent semantic redundancy; and 3) Uncertain channel estimation. To address these issues, we propose a Generative AI Model assisted 3D SC (GAM-3DSC) system. Firstly, we introduce a 3D Semantic Extractor (3DSE), which employs generative AI models, including Segment Anything Model (SAM) and Neural Radiance Field (NeRF), to extract key semantics from a 3D scenario based on user requirements. The extracted 3D semantics are represented as multi-perspective images of the goal-oriented 3D object. Then, we present an Adaptive Semantic Compression Model (ASCM) for encoding these multi-perspective images, in which we use a semantic encoder with two output heads to perform semantic encoding and mask redundant semantics in the latent semantic space, respectively. Next, we design a conditional Generative adversarial network and Diffusion model aided-Channel Estimation (GDCE) to estimate and refine the Channel State Information (CSI) of physical channels. Finally, simulation results demonstrate the advantages of the proposed GAM-3DSC system in effectively transmitting the goal-oriented 3D scenario.