A Preliminary Roadmap for LLMs as Assistants in Exploring, Analyzing, and Visualizing Knowledge Graphs

TL;DR

This study explores how large language models can assist users in exploring, analyzing, and visualizing knowledge graphs through user interviews, identifying key functionalities, user preferences, and challenges to inform future system design.

Contribution

It provides a preliminary roadmap for designing LLM-driven knowledge graph exploration tools based on user insights and identifies future opportunities in this emerging area.

Findings

Users want LLMs to facilitate data retrieval and relationship identification in KGs.

Participants prefer chat-based interfaces with visual aids for interaction.

Concerns include semantic accuracy, hallucinations, and prompt engineering challenges.

Abstract

We present a mixed-methods study to explore how large language models (LLMs) can assist users in the visual exploration and analysis of knowledge graphs (KGs). We surveyed and interviewed 20 professionals from industry, government laboratories, and academia who regularly work with KGs and LLMs, either collaboratively or concurrently. Our findings show that participants overwhelmingly want an LLM to facilitate data retrieval from KGs through joint query construction, to identify interesting relationships in the KG through multi-turn conversation, and to create on-demand visualizations from the KG that enhance their trust in the LLM's outputs. To interact with an LLM, participants strongly prefer a chat-based 'widget,' built on top of their regular analysis workflows, with the ability to guide the LLM using their interactions with a visualization. When viewing an LLM's outputs, participants similarly prefer a combination of annotated visuals (e.g., subgraphs or tables extracted from the KG) alongside summarizing text. However, participants also expressed concerns about an LLM's ability to maintain semantic intent when translating natural language questions into KG queries, the risk of an LLM 'hallucinating' false data from the KG, and the difficulties of engineering a 'perfect prompt.' From the analysis of our interviews, we contribute a preliminary roadmap for the design of LLM-driven knowledge graph exploration systems and outline future opportunities in this emergent design space.