Large Generative Model-assisted Talking-face Semantic Communication System

TL;DR

This paper presents a novel large generative model-assisted talking-face semantic communication system that enhances bandwidth efficiency, semantic clarity, and video quality in talking-face video transmission.

Contribution

It introduces a comprehensive LGM-TSC system with a generative semantic extractor, a knowledge base for disambiguation, and a semantic reconstructor, advancing semantic communication for talking-face videos.

Findings

High information density in semantic extraction

Effective semantic disambiguation using a knowledge base

High-quality talking-face video reconstruction

Abstract

The rapid development of generative Artificial Intelligence (AI) continually unveils the potential of Semantic Communication (SemCom). However, current talking-face SemCom systems still encounter challenges such as low bandwidth utilization, semantic ambiguity, and diminished Quality of Experience (QoE). This study introduces a Large Generative Model-assisted Talking-face Semantic Communication (LGM-TSC) System tailored for the talking-face video communication. Firstly, we introduce a Generative Semantic Extractor (GSE) at the transmitter based on the FunASR model to convert semantically sparse talking-face videos into texts with high information density. Secondly, we establish a private Knowledge Base (KB) based on the Large Language Model (LLM) for semantic disambiguation and correction, complemented by a joint knowledge base-semantic-channel coding scheme. Finally, at the receiver, we propose a Generative Semantic Reconstructor (GSR) that utilizes BERT-VITS2 and SadTalker models to transform text back into a high-QoE talking-face video matching the user's timbre. Simulation results demonstrate the feasibility and effectiveness of the proposed LGM-TSC system.