The Evolution of LLM Adoption in Industry Data Curation Practices

TL;DR

This paper examines how industry practitioners are increasingly adopting large language models (LLMs) in data curation, shifting workflows towards insights-first approaches and integrating LLM-generated datasets to handle complex unstructured data.

Contribution

It provides empirical insights into evolving LLM adoption strategies, usage scenarios, and the shift towards insights-first workflows in industry data curation practices.

Findings

Shift from heuristic to insights-first workflows supported by LLMs

Use of LLM-generated 'silver' and 'super golden' datasets alongside traditional datasets

Growing integration of LLMs in large-scale unstructured data analysis

Abstract

As large language models (LLMs) grow increasingly adept at processing unstructured text data, they offer new opportunities to enhance data curation workflows. This paper explores the evolution of LLM adoption among practitioners at a large technology company, evaluating the impact of LLMs in data curation tasks through participants' perceptions, integration strategies, and reported usage scenarios. Through a series of surveys, interviews, and user studies, we provide a timely snapshot of how organizations are navigating a pivotal moment in LLM evolution. In Q2 2023, we conducted a survey to assess LLM adoption in industry for development tasks (N=84), and facilitated expert interviews to assess evolving data needs (N=10) in Q3 2023. In Q2 2024, we explored practitioners' current and anticipated LLM usage through a user study involving two LLM-based prototypes (N=12). While each study addressed distinct research goals, they revealed a broader narrative about evolving LLM usage in aggregate. We discovered an emerging shift in data understanding from heuristic-first, bottom-up approaches to insights-first, top-down workflows supported by LLMs. Furthermore, to respond to a more complex data landscape, data practitioners now supplement traditional subject-expert-created 'golden datasets' with LLM-generated 'silver' datasets and rigorously validated 'super golden' datasets curated by diverse experts. This research sheds light on the transformative role of LLMs in large-scale analysis of unstructured data and highlights opportunities for further tool development.