Embracing Radiance Field Rendering in 6G: Over-the-Air Training and Inference with 3D Contents

TL;DR

This paper explores integrating neural radiance fields and 3D Gaussian splatting into 6G networks, focusing on over-the-air training and rendering techniques to support immersive 3D applications.

Contribution

It provides a comprehensive overview of radiance field rendering integration in 6G, including training, rendering architectures, and communication strategies, with novel federated learning and semantic communication approaches.

Findings

Proposes hierarchical federated learning for large-scale 3D model training.

Introduces three practical over-the-air rendering architectures.

Develops model compression and acceleration techniques for efficient transmission.

Abstract

The efficient representation, transmission, and reconstruction of three-dimensional (3D) contents are becoming increasingly important for sixth-generation (6G) networks that aim to merge virtual and physical worlds for offering immersive communication experiences. Neural radiance field (NeRF) and 3D Gaussian splatting (3D-GS) have recently emerged as two promising 3D representation techniques based on radiance field rendering, which are able to provide photorealistic rendering results for complex scenes. Therefore, embracing NeRF and 3D-GS in 6G networks is envisioned to be a prominent solution to support emerging 3D applications with enhanced quality of experience. This paper provides a comprehensive overview on the integration of NeRF and 3D-GS in 6G. First, we review the basics of the radiance field rendering techniques, and highlight their applications and implementation challenges over wireless networks. Next, we consider the over-the-air training of NeRF and 3D-GS models over wireless networks by presenting various learning techniques. We particularly focus on the federated learning design over a hierarchical device-edge-cloud architecture, which is suitable for exploiting distributed data and computing resources over 6G networks to train large models representing large-scale scenes. Then, we consider the over-the-air rendering of NeRF and 3D-GS models at wireless network edge. We present three practical rendering architectures, namely local, remote, and co-rendering, respectively, and provide model compression approaches to facilitate the transmission of radiance field models for rendering. We also present rendering acceleration approaches and joint computation and communication designs to enhance the rendering efficiency. In a case study, we propose a new semantic communication enabled 3D content transmission design.