MultiSurf-GPT: Facilitating Context-Aware Reasoning with Large-Scale Language Models for Multimodal Surface Sensing

TL;DR

MultiSurf-GPT leverages GPT-4o to process diverse multimodal surface sensing data, enabling context-aware analytics in health, manufacturing, and safety applications with promising preliminary results.

Contribution

Introduces MultiSurf-GPT, a novel framework that uses large-scale language models to interpret multiple sensing modalities for enhanced context-aware reasoning.

Findings

Successfully identified low-level information from multimodal data

Inferred high-level context-aware analytics

Demonstrated potential for faster, cost-effective application development

Abstract

Surface sensing is widely employed in health diagnostics, manufacturing and safety monitoring. Advances in mobile sensing affords this potential for context awareness in mobile computing, typically with a single sensing modality. Emerging multimodal large-scale language models offer new opportunities. We propose MultiSurf-GPT, which utilizes the advanced capabilities of GPT-4o to process and interpret diverse modalities (radar, microscope and multispectral data) uniformly based on prompting strategies (zero-shot and few-shot prompting). We preliminarily validated our framework by using MultiSurf-GPT to identify low-level information, and to infer high-level context-aware analytics, demonstrating the capability of augmenting context-aware insights. This framework shows promise as a tool to expedite the development of more complex context-aware applications in the future, providing a faster, more cost-effective, and integrated solution.