Creating User-steerable Projections with Interactive Semantic Mapping

TL;DR

This paper presents a user-guided projection method that uses natural language prompts and multimodal models to create interpretable, customizable visualizations of high-dimensional data, enhancing exploratory data analysis.

Contribution

It introduces a novel interactive framework that allows users to steer dimensionality reduction with natural language, enabling semantic control beyond data-driven features.

Findings

Improves cluster separation in visualizations.

Enables dynamic, user-driven data exploration.

Bridges automated DR and human-centered analysis.

Abstract

Dimensionality reduction (DR) techniques map high-dimensional data into lower-dimensional spaces. Yet, current DR techniques are not designed to explore semantic structure that is not directly available in the form of variables or class labels. We introduce a novel user-guided projection framework for image and text data that enables customizable, interpretable, data visualizations via zero-shot classification with Multimodal Large Language Models (MLLMs). We enable users to steer projections dynamically via natural-language guiding prompts, to specify high-level semantic relationships of interest to the users which are not explicitly present in the data dimensions. We evaluate our method across several datasets and show that it not only enhances cluster separation, but also transforms DR into an interactive, user-driven process. Our approach bridges the gap between fully automated DR techniques and human-centered data exploration, offering a flexible and adaptive way to tailor projections to specific analytical needs.