Results-Actionability Gap: Understanding How Practitioners Evaluate LLM Products in the Wild

TL;DR

This paper explores how practitioners evaluate LLM products in real-world settings, revealing a gap between gathering evaluation data and translating it into actionable improvements, and offers strategies to bridge this gap.

Contribution

It introduces the concept of the results-actionability gap and provides practical strategies to help practitioners move from informal to systematic evaluation practices.

Findings

Practitioners use diverse evaluation practices, from vibe checks to meta-work.

A new 'results-actionability gap' exists where data is gathered but not translated into improvements.

Successful teams employ strategies to formalize evaluation and bridge this gap.

Abstract

How do product teams evaluate LLM-powered products? As organizations integrate large language models (LLMs) into digital products, their unpredictable nature makes traditional evaluation approaches inadequate, yet little is known about how practitioners navigate this challenge. Through interviews with nineteen practitioners across diverse sectors, we identify ten evaluation practices spanning informal 'vibe checks' to organizational meta-work. Beyond confirming four documented challenges, we introduce a novel fifth we call the results-actionability gap, in which practitioners gather evaluation data but cannot translate findings into concrete improvements. Drawing on patterns from successful teams, we contribute strategies to bridge this gap, supporting practitioners' formalization journey from ad-hoc interpretive practices (e.g., vibe checks) toward systematic evaluation. Our analysis suggests these interpretive practices are necessary adaptations to LLM characteristics rather than methodological failures. For HCI researchers, this presents a research opportunity to support practitioners in systematizing emerging practices rather than developing new evaluation frameworks.