Prompts Are Programs Too! Understanding How Developers Build Software Containing Prompts

TL;DR

This paper explores prompt programming as a new form of software development where developers craft prompts for AI models, revealing unique practices, challenges, and the need for specialized tools.

Contribution

It introduces the concept of prompt programming, provides empirical insights from developer interviews, and highlights differences from traditional coding practices.

Findings

Prompt programmers develop mental models of the foundation model's behavior.

Prompt programming differs significantly from traditional software development.

Developers struggle to build reliable mental models despite experience.

Abstract

Generative pre-trained models power intelligent software features used by millions of users controlled by developer-written natural language prompts. Despite the impact of prompt-powered software, little is known about its development process and its relationship to programming. In this work, we argue that some prompts are programs and that the development of prompts is a distinct phenomenon in programming known as "prompt programming". We develop an understanding of prompt programming using Straussian grounded theory through interviews with 20 developers engaged in prompt development across a variety of contexts, models, domains, and prompt structures. We contribute 15 observations to form a preliminary understanding of current prompt programming practices. For example, rather than building mental models of code, prompt programmers develop mental models of the foundation model (FM)'s behavior on the prompt by interacting with the FM. While prior research shows that experts have well-formed mental models, we find that prompt programmers who have developed dozens of prompts still struggle to develop reliable mental models. Our observations show that prompt programming differs from traditional software development, motivating the creation of prompt programming tools and providing implications for software engineering stakeholders.