Key Considerations for Domain Expert Involvement in LLM Design and Evaluation: An Ethnographic Study

TL;DR

This ethnographic study explores how development teams involving domain experts in LLM design and evaluation navigate practical challenges, revealing key practices and proposing design opportunities for more effective workflows.

Contribution

It provides an in-depth ethnographic analysis of real-world practices and challenges in involving domain experts in LLM development, highlighting new strategies and design considerations.

Findings

Teams create workarounds for data collection

Use augmentation when expert input is limited

Co-develop evaluation criteria with experts

Abstract

Large Language Models (LLMs) are increasingly developed for use in complex professional domains, yet little is known about how teams design and evaluate these systems in practice. This paper examines the challenges and trade-offs in LLM development through a 12-week ethnographic study of a team building a pedagogical chatbot. The researcher observed design and evaluation activities and conducted interviews with both developers and domain experts. Analysis revealed four key practices: creating workarounds for data collection, turning to augmentation when expert input was limited, co-developing evaluation criteria with experts, and adopting hybrid expert-developer-LLM evaluation strategies. These practices show how teams made strategic decisions under constraints and demonstrate the central role of domain expertise in shaping the system. Challenges included expert motivation and trust, difficulties structuring participatory design, and questions around ownership and integration of expert knowledge. We propose design opportunities for future LLM development workflows that emphasize AI literacy, transparent consent, and frameworks recognizing evolving expert roles.