Semantic Communications for 3D Human Face Transmission with Neural Radiance Fields

TL;DR

This paper introduces NeRF-SeCom, a novel framework combining neural radiance fields and semantic communication to efficiently transmit 3D human face data, achieving high-quality visualization with reduced communication overhead.

Contribution

The paper presents a new NeRF-based semantic communication framework for 3D face transmission, including a static/dynamic feature classification and a feature prediction mechanism.

Findings

Significantly improves 3D face visualization quality

Reduces communication overhead compared to benchmarks

Effective real-time 3D face reconstruction

Abstract

This paper investigates the transmission of three-dimensional (3D) human face content for immersive communication over a rate-constrained transmitter-receiver link. We propose a new framework named NeRF-SeCom, which leverages neural radiance fields (NeRF) and semantic communications to improve the quality of 3D visualizations while minimizing the communication overhead. In the NeRF-SeCom framework, we first train a NeRF face model based on the NeRFBlendShape method, which is pre-shared between the transmitter and receiver as the semantic knowledge base to facilitate the real-time transmission. Next, with knowledge base, the transmitter extracts and sends only the essential semantic features for the receiver to reconstruct 3D face in real time. To optimize the transmission efficiency, we classify the expression features into static and dynamic types. Over each video chunk, static features are transmitted once for all frames, whereas dynamic features are transmitted over a portion of frames to adhere to rate constraints. Additionally, we propose a feature prediction mechanism, which allows the receiver to predict the dynamic features for frames that are not transmitted. Experiments show that our proposed NeRF-SeCom framework significantly outperforms benchmark methods in delivering high-quality 3D visualizations of human faces.