What You Prompt is What You Get: Increasing Transparency of Prompting Using Prompt Cards

TL;DR

This paper introduces prompt cards as a structured documentation tool to enhance transparency, reproducibility, and evaluation of prompt engineering practices in large language models.

Contribution

It proposes the prompt card approach inspired by model cards, demonstrating its use in a specific task to improve prompt methodology and transparency.

Findings

Prompt cards improve reproducibility of prompt engineering.

They enhance transparency and systematic documentation.

Prompt cards serve as an effective alternative to benchmarking.

Abstract

The rapid advancement and impressive capabilities of large language models (LLMs) have given rise to the field of prompt engineering, the practice of crafting inputs to guide LLMs toward high-quality, task-relevant outputs. A critical challenge facing the field is the lack of standardised prompt documentation and evaluation practices. Prompts can be long, complex and difficult to evaluate on subjective tasks. To address this challenge, we propose the use of prompt cards, structured summaries of prompt engineering practices inspired by the concept of model cards. Through prompt cards, the specific goals, considerations and steps taken during prompt engineering can be systematically documented and assessed. We present the prompt card approach and illustrate it on a specific task called wordalisation, in which structured numerical data is transformed into text. We argue that a well-structured prompt card can enable better reproducibility, transparency, improve prompt methodology and give an effective alternative to benchmarking for judging the quality of generated texts. By systemically capturing underlying model details, prompt intent, contextualisation strategies, evaluation practices and ethical considerations, prompt cards make explicit the often implicit design decisions that shape system behaviour. Documenting these choices is important as prompting increasingly involves complex pipelines with multiple moving parts.