BundleFit: Display and See-Through Models for Augmented Reality Head-Mounted Displays

TL;DR

BundleFit introduces a novel optical modeling approach for AR head-mounted displays that balances accuracy, efficiency, and accessibility by fitting ray bundles outside the hardware, validated through simulations and real-world tests.

Contribution

The paper proposes a new bundle-fit-based model that improves optical accuracy and accessibility over existing black-box and white-box models for AR displays.

Findings

Effective in modeling display and see-through optics

Achieves high accuracy with runtime optimization

Validated on simulations and real hardware

Abstract

The head-mounted display is a vital component of augmented reality, incorporating optics with complex display and see-through optical behavior. Computationally modeling these optical behaviors requires meeting three key criteria: accuracy, efficiency, and accessibility. In recent years, various approaches have been proposed to model display and see-through optics, which can broadly be classified into black-box and white-box models. However, both categories face significant limitations that hinder their adoption in commercial applications. To overcome these challenges, we leveraged prior knowledge of ray bundle properties outside the optical hardware and proposed a novel bundle-fit-based model. In this approach, the ray paths within the optics are treated as a black box, while a lightweight optimization problem is solved to fit the ray bundle outside the optics. This method effectively addresses the accuracy issues of black-box models and the accessibility challenges of white-box models. Although our model involves runtime optimization, this is typically not a concern, as it can use the solution from a previous query to initialize the optimization for the current query. We evaluated the performance of our proposed method through both simulations and experiments on real hardware, demonstrating its effectiveness.