A Smart-Glasses for Emergency Medical Services via Multimodal Multitask Learning

TL;DR

EMSGlass is a multimodal smart-glasses system powered by EMSNet and EMSServe, enabling real-time understanding and decision support for EMTs in emergency scenarios, improving accuracy, speed, and operational efficiency.

Contribution

This paper introduces EMSNet, a novel multimodal multitask model, and EMSServe, a low-latency inference framework, specifically designed for emergency medical services using smart-glasses.

Findings

EMSNet supports up to five EMS tasks with superior accuracy.

EMSServe achieves up to 11.7x speedup over standard inference methods.

User study shows improved situational awareness and decision-making for EMTs.

Abstract

Emergency Medical Technicians (EMTs) operate in high-pressure environments, making rapid, life-critical decisions under heavy cognitive and operational loads. We present EMSGlass, a smart-glasses system powered by EMSNet, the first multimodal multitask model for Emergency Medical Services (EMS), and EMSServe, a low-latency multimodal serving framework tailored to EMS scenarios. EMSNet integrates text, vital signs, and scene images to construct a unified real-time understanding of EMS incidents. Trained on real-world multimodal EMS datasets, EMSNet simultaneously supports up to five critical EMS tasks with superior accuracy compared to state-of-the-art unimodal baselines. Built on top of PyTorch, EMSServe introduces a modality-aware model splitter and a feature caching mechanism, achieving adaptive and efficient inference across heterogeneous hardware while addressing the challenge of asynchronous modality arrival in the field. By optimizing multimodal inference execution in EMS scenarios, EMSServe achieves 1.9x -- 11.7x speedup over direct PyTorch multimodal inference. A user study evaluation with six professional EMTs demonstrates that EMSGlass enhances real-time situational awareness, decision-making speed, and operational efficiency through intuitive on-glass interaction. In addition, qualitative insights from the user study provide actionable directions for extending EMSGlass toward next-generation AI-enabled EMS systems, bridging multimodal intelligence with real-world emergency response workflows.