Rate-Utility Optimized Streaming of Volumetric Media for Augmented Reality

TL;DR

This paper presents a novel system for streaming volumetric media in AR that reduces bandwidth and latency through 3D tiling, view-dependent culling, and an optimal rate-utility allocation algorithm, enhancing user experience.

Contribution

It introduces the first dedicated volumetric media streaming system with view-aware tiling, a window-based buffer for low latency, and a provably optimal rate-utility optimization algorithm.

Findings

System reduces bandwidth via 3D tiling and culling.

Latency is minimized with a window-based buffer.

Simulation shows improved quality and user experience.

Abstract

Volumetric media, popularly known as holograms, need to be delivered to users using both on-demand and live streaming, for new augmented reality (AR) and virtual reality (VR) experiences. As in video streaming, hologram streaming must support network adaptivity and fast startup, but must also moderate large bandwidths, multiple simultaneously streaming objects, and frequent user interaction, which requires low delay. In this paper, we introduce the first system to our knowledge designed specifically for streaming volumetric media. The system reduces bandwidth by introducing 3D tiles, and culling them or reducing their level of detail depending on their relation to the user's view frustum and distance to the user. Our system reduces latency by introducing a window-based buffer, which in contrast to a queue-based buffer allows insertions near the head of the buffer rather than only at the tail of the buffer, to respond quickly to user interaction. To allocate bits between different tiles across multiple objects, we introduce a simple greedy yet provably optimal algorithm for rate-utility optimization. We introduce utility measures based not only on the underlying quality of the representation, but on the level of detail relative to the user's viewpoint and device resolution. Simulation results show that the proposed algorithm provides superior quality compared to existing video-streaming approaches adapted to hologram streaming, in terms of utility and user experience over variable, throughput-constrained networks.