Can Foundation Models Revolutionize Mobile AR Sparse Sensing?

TL;DR

This paper explores how foundation models can enhance mobile AR sparse sensing by improving accuracy and scalability, especially in geometry-aware image warping and 3D scene reconstruction, addressing longstanding trade-offs.

Contribution

It demonstrates the potential of foundation models to significantly improve mobile sparse sensing accuracy and scalability, particularly in geometry-aware image warping and 3D reconstruction.

Findings

Foundation models improve geometry-aware image warping accuracy.

They enable scalable sparse sensing for 3D scene reconstruction.

The study highlights open challenges in integrating foundation models into mobile sensing.

Abstract

Mobile sensing systems have long faced a fundamental trade-off between sensing quality and efficiency due to constraints in computation, power, and other limitations. Sparse sensing, which aims to acquire and process only a subset of sensor data, has been a key strategy for maintaining performance under such constraints. However, existing sparse sensing methods often suffer from reduced accuracy, as missing information across space and time introduces uncertainty into many sensing systems. In this work, we investigate whether foundation models can change the landscape of mobile sparse sensing. Using real-world mobile AR data, our evaluations demonstrate that foundation models offer significant improvements in geometry-aware image warping, a central technique for enabling accurate reuse of cross-frame information. Furthermore, our study demonstrates the scalability of foundation model-based sparse sensing and shows its leading performance in 3D scene reconstruction. Collectively, our study reveals critical aspects of the promises and the open challenges of integrating foundation models into mobile sparse sensing systems.