Ground-Truth Depth in Vision Language Models: Spatial Context Understanding in Conversational AI for XR-Robotic Support in Emergency First Response

TL;DR

This paper demonstrates that integrating depth sensing with vision language models enhances spatial reasoning and situational awareness in emergency first response scenarios, leading to more accurate and stable distance estimations without increasing workload.

Contribution

It introduces a prototype that fuses depth sensing with vision language models to provide metrically grounded spatial information for emergency response tasks.

Findings

Depth-augmented VLM improves distance estimation accuracy.

Depth-augmentation raises situational awareness without increasing workload.

Depth-agnostic assistance can increase workload and reduce accuracy.

Abstract

Large language models (LLMs) are increasingly used in emergency first response (EFR) applications to support situational awareness (SA) and decision-making, yet most operate on text or 2D imagery and offer little support for core EFR SA competencies like spatial reasoning. We address this gap by evaluating a prototype that fuses robot-mounted depth sensing and YOLO detection with a vision language model (VLM) capable of verbalizing metrically-grounded distances of detected objects (e.g., the chair is 3.02 meters away). In a mixed-reality toxic-smoke scenario, participants estimated distances to a victim and an exit window under three conditions: video-only, depth-agnostic VLM, and depth-augmented VLM. Depth-augmentation improved objective accuracy and stability, e.g., the victim and window distance estimation error dropped, while raising situational awareness without increasing workload. Conversely, depth- agnostic assistance increased workload and slightly worsened accuracy. We contribute to human SA augmentation by demonstrating that metrically grounded, object-centric verbal information supports spatial reasoning in EFR and improves decision-relevant judgments under time pressure.