MotionTrace: IMU-based Field of View Prediction for Smartphone AR Interactions

TL;DR

MotionTrace predicts smartphone user’s field of view in AR by estimating future hand and phone positions using inertial sensors, improving streaming performance and user experience.

Contribution

This paper introduces MotionTrace, a novel inertial sensor-based method for predicting future smartphone positions to enhance AR streaming efficiency.

Findings

Achieves average MSE between 0.11 and 143.62 mm across various time horizons.

Effectively predicts future phone position for up to 800ms ahead.

Validated on large motion capture and smartphone datasets.

Abstract

For handheld smartphone AR interactions, bandwidth is a critical constraint. Streaming techniques have been developed to provide a seamless and high-quality user experience despite these challenges. To optimize streaming performance in smartphone-based AR, accurate prediction of the user's field of view is essential. This prediction allows the system to prioritize loading digital content that the user is likely to engage with, enhancing the overall interactivity and immersion of the AR experience. In this paper, we present MotionTrace, a method for predicting the user's field of view using a smartphone's inertial sensor. This method continuously estimates the user's hand position in 3D-space to localize the phone position. We evaluated MotionTrace over future hand positions at 50, 100, 200, 400, and 800ms time horizons using the large motion capture (AMASS) and smartphone-based full-body pose estimation (Pose-on-the-Go) datasets. We found that our method can estimate the future phone position of the user with an average MSE between 0.11 - 143.62 mm across different time horizons.