From 'Here' to 'There': Exploring Proximity Semantics in Multimodal Data Exploration

TL;DR

This paper introduces a multimodal data exploration system that combines sketching, language, and annotations, leveraging proximity semantics to improve pattern discovery in complex datasets.

Contribution

It presents a novel hybrid architecture and conceptual framework for understanding proximity semantics in multimodal interactions, supported by a user study.

Findings

Participants used spatial, temporal, and visual proximity to disambiguate meaning.

Proximity semantics influence how users relate multimodal elements during exploration.

The system supports pattern matching and semantic constraints through hybrid architecture.

Abstract

Modern data exploration tools often struggle to capture the subtleties of analytical intent, especially when users seek patterns that are difficult to specify using traditional query methods or natural language alone. We introduce a multimodal research probe for querying time-series and geospatial data that integrates free-form sketching, natural language, and visual annotations within a unified interaction space. Users articulate queries by sketching trends or spatial paths and augmenting them with annotations and analytical directives grounded in shared spatial and temporal context. The system employs a hybrid architecture combining geometric sketch matching and visual language models (VLMs) to support queries that interleave pattern matching and semantic constraints. Through a preliminary study with 20 participants, we observed recurring interaction patterns in which participants used spatial, temporal, and visual proximity to relate sketches, annotations, and language. Rather than treating these as isolated inputs, participants relied on their relative placement to disambiguate meaning. We analyze these behaviors as evidence for proximity semantics (PS), a form of deictic disambiguation in which meaning is shaped by the closeness of multimodal elements within a shared interaction space. We present PS as a conceptual lens grounded in observed user behavior, and discuss its implications for the design of future multimodal data exploration systems.