Communication Efficient Robotic Mixed Reality with Gaussian Splatting Cross-Layer Optimization

TL;DR

This paper introduces a novel Gaussian splatting-based framework for robotic mixed reality that significantly reduces communication costs by optimizing content and power allocation, enabling efficient, photo-realistic remote robot visualization.

Contribution

It proposes the GSMR system with a cross-layer optimization framework GSCLO and an accelerated penalty optimization algorithm, advancing low-cost, high-quality RoboMR communication.

Findings

GSMR achieves ultra-low communication costs in RoboMR.

GSCLO effectively optimizes content switching and power allocation.

Experiments show significant performance improvements across various scenarios.

Abstract

Realizing low-cost communication in robotic mixed reality (RoboMR) systems presents a challenge, due to the necessity of uploading high-resolution images through wireless channels. This paper proposes Gaussian splatting (GS) RoboMR (GSMR), which enables the simulator to opportunistically render a photo-realistic view from the robot's pose by calling ``memory'' from a GS model, thus reducing the need for excessive image uploads. However, the GS model may involve discrepancies compared to the actual environments. To this end, a GS cross-layer optimization (GSCLO) framework is further proposed, which jointly optimizes content switching (i.e., deciding whether to upload image or not) and power allocation (i.e., adjusting to content profiles) across different frames by minimizing a newly derived GSMR loss function. The GSCLO problem is addressed by an accelerated penalty optimization (APO) algorithm that reduces computational complexity by over $10$x compared to traditional branch-and-bound and search algorithms. Moreover, variants of GSCLO are presented to achieve robust, low-power, and multi-robot GSMR. Extensive experiments demonstrate that the proposed GSMR paradigm and GSCLO method achieve significant improvements over existing benchmarks on both wheeled and legged robots in terms of diverse metrics in various scenarios. For the first time, it is found that RoboMR can be achieved with ultra-low communication costs, and mixture of data is useful for enhancing GS performance in dynamic scenarios.