Data Discovery using LLMs -- A Study of Data User Behaviour

TL;DR

This study investigates how researchers interact with large language models for data discovery, revealing that LLMs are primarily tools rather than conversational partners, with user experience influenced by prior LLM familiarity.

Contribution

It provides empirical insights into researcher behavior with LLMs in data search, highlighting the impact of persona prompting and user experience levels.

Findings

Researchers interact naturally with LLMs for data search.

LLMs are used mainly as tools, not conversational partners.

Persona prompting influences experienced users' experience.

Abstract

Data search for scientific research is more complex than a simple web search. The emergence of large language models (LLMs) and their applicability for scientific tasks offers new opportunities for researchers who are looking for data, e.g., to freely express their data needs instead of fitting them into restrictions of data catalogues and portals. However, this also creates uncertainty about whether LLMs are suitable for this task. To answer this question, we conducted a user study with 32 researchers. We qualitatively and quantitively analysed participants' information interaction behaviour while searching for data using LLMs in two data search tasks, one in which we prompted the LLM to behave as a persona. We found that participants interact with LLMs in natural language, but LLMs remain a tool for them rather than an equal conversational partner. This changes slightly when the LLM is prompted to behave as a persona, but the prompting only affects participants' user experience when they are already experienced in LLM use.