Kalman Filter-based Head Motion Prediction for Cloud-based Mixed Reality

TL;DR

This paper proposes a Kalman filter-based method to predict head motion in cloud-based volumetric video streaming for mixed reality, aiming to reduce latency and improve registration accuracy.

Contribution

It introduces a Kalman filter approach for head motion prediction and compares its performance to autoregression models in MR streaming systems.

Findings

Kalman filter predicts head orientations 0.5 degrees more accurately at 60 ms look-ahead.

The approach reduces latency-induced registration errors in MR environments.

Performance analysis using recorded head motion traces demonstrates improved accuracy.

Abstract

Volumetric video allows viewers to experience highly-realistic 3D content with six degrees of freedom in mixed reality (MR) environments. Rendering complex volumetric videos can require a prohibitively high amount of computational power for mobile devices. A promising technique to reduce the computational burden on mobile devices is to perform the rendering at a cloud server. However, cloud-based rendering systems suffer from an increased interaction (motion-to-photon) latency that may cause registration errors in MR environments. One way of reducing the effective latency is to predict the viewer's head pose and render the corresponding view from the volumetric video in advance. In this paper, we design a Kalman filter for head motion prediction in our cloud-based volumetric video streaming system. We analyze the performance of our approach using recorded head motion traces and compare its performance to an autoregression model for different prediction intervals (look-ahead times). Our results show that the Kalman filter can predict head orientations 0.5 degrees more accurately than the autoregression model for a look-ahead time of 60 ms.