GIST: Multimodal Knowledge Extraction and Spatial Grounding via Intelligent Semantic Topology

TL;DR

GIST is a multimodal knowledge extraction system that creates a semantic spatial topology from mobile point clouds, enabling advanced navigation and interaction tasks in complex environments.

Contribution

It introduces a novel pipeline that transforms point clouds into a semantic topology, improving spatial understanding and human-AI interaction in cluttered spaces.

Findings

Achieved a 1.04 m top-5 mean translation error in localizing semantic zones.

Outperformed sequence-based instruction generation baselines in LLM evaluations.

Reached an 80% navigation success rate using verbal cues in real-world tests.

Abstract

Navigating complex, densely packed environments like retail stores, warehouses, and hospitals poses a significant spatial grounding challenge for humans and embodied AI. In these spaces, dense visual features quickly become stale given the quasi-static nature of items, and long-tail semantic distributions challenge traditional computer vision. While Vision-Language Models (VLMs) help assistive systems navigate semantically-rich spaces, they still struggle with spatial grounding in cluttered environments. We present GIST (Grounded Intelligent Semantic Topology), a multimodal knowledge extraction pipeline that transforms a consumer-grade mobile point cloud into a semantically annotated navigation topology. Our architecture distills the scene into a 2D occupancy map, extracts its topological layout, and overlays a lightweight semantic layer via intelligent keyframe and semantic selection. We demonstrate the versatility of this structured spatial knowledge through critical downstream Human-AI interaction tasks: (1) an intent-driven Semantic Search engine that actively infers categorical alternatives and zones when exact matches fail; (2) a one-shot Semantic Localizer achieving a 1.04 m top-5 mean translation error; (3) a Zone Classification module that segments the walkable floor plan into high-level semantic regions; and (4) a Visually-Grounded Instruction Generator that synthesizes optimal paths into egocentric, landmark-rich natural language routing. In multi-criteria LLM evaluations, GIST outperforms sequence-based instruction generation baselines. Finally, an in-situ formative evaluation (N=5) yields an 80% navigation success rate relying solely on verbal cues, validating the system's capacity for universal design.