Accelerating Physical Property Reasoning for Augmented Visual Cognition

TL;DR

This paper presents \\sysname, a system that significantly accelerates vision-guided physical property reasoning, enabling real-time augmented visual cognition with high accuracy and robustness in real-world environments.

Contribution

sysname introduces a set of algorithmic and systematic optimizations that reduce reasoning latency from minutes to seconds, enabling practical real-time physical property inference.

Findings

Achieves 62.987.2 speedup over baselines.

Maintains comparable or better physical property estimation accuracy.

Demonstrates robust performance in real-world scenarios with fewer views.

Abstract

This paper introduces \sysname, a system that accelerates vision-guided physical property reasoning to enable augmented visual cognition. \sysname minimizes the run-time latency of this reasoning pipeline through a combination of both algorithmic and systematic optimizations, including rapid geometric 3D reconstruction, efficient semantic feature fusion, and parallel view encoding. Through these simple yet effective optimizations, \sysname reduces the end-to-end latency of this reasoning pipeline from 10--20 minutes to less than 6 seconds. A head-to-head comparison on the ABO dataset shows that \sysname achieves this 62.9$\times$--287.2$\times$ speedup while not only reaching on-par (and sometimes slightly better) object-level physical property estimation accuracy(e.g. mass), but also demonstrating superior performance in material segmentation and voxel-level inference than two SOTA baselines. We further combine gaze-tracking with \sysname to localize the object of interest in cluttered, real-world environments, streamlining the physical property reasoning on smart glasses. The case study with Meta Aria Glasses conducted at an IKEA furniture store demonstrates that \sysname achives consistently high performance compared to controlled captures, providing robust property estimations even with fewer views in real-world scenarios.