Themes of Building LLM-based Applications for Production: A Practitioner's View

TL;DR

This paper systematically maps practitioner discussions on deploying LLMs in production, highlighting key themes like design, architecture, capabilities, infrastructure, and ethical challenges to guide future development.

Contribution

It provides a comprehensive overview of practical considerations and challenges in deploying LLM-based applications, based on analysis of online practitioner discussions.

Findings

Retrieval-augmented generation (RAG) is a prevalent design focus.

Model capabilities and enhancement techniques are frequently discussed.

Infrastructure, tooling, and ethical challenges are key concerns.

Abstract

Background: Large language models (LLMs) have become a paramount interest of researchers and practitioners alike, yet a comprehensive overview of key considerations for those developing LLM-based systems is lacking. This study addresses this gap by collecting and mapping the topics practitioners discuss online, offering practical insights into where priorities lie in developing LLM-based applications. Method: We collected 189 videos from 2022 to 2024 from practitioners actively developing such systems and discussing various aspects they encounter during development and deployment of LLMs in production. We analyzed the transcripts using BERTopic, then manually sorted and merged the generated topics into themes, leading to a total of 20 topics in 8 themes. Results: The most prevalent topics fall within the theme Design & Architecture, with a strong focus on retrieval-augmented generation (RAG) systems. Other frequently discussed topics include model capabilities and enhancement techniques (e.g., fine-tuning, prompt engineering), infrastructure and tooling, and risks and ethical challenges. Implications: Our results highlight current discussions and challenges in deploying LLMs in production. This way, we provide a systematic overview of key aspects practitioners should be aware of when developing LLM-based applications. We further pale off topics of interest for academics where further research is needed.