The Effect of Education in Prompt Engineering: Evidence from Journalists

TL;DR

This study investigates how prompt engineering training affects journalists' interactions with large language models, revealing mixed impacts on perceived expertise, text accuracy, and reader perception across different dimensions.

Contribution

It provides empirical evidence on the effects of prompt engineering training on user experience, text quality, and perception in a real-world journalistic context.

Findings

Training increased perceived journalist expertise

Training decreased perceived helpfulness of LLMs

Effects on accuracy varied by task difficulty

Abstract

Large language models (LLMs) are increasingly used in daily work. In this paper, we analyze whether training in prompt engineering can improve the interactions of users with LLMs. For this, we conducted a field experiment where we asked journalists to write short texts before and after training in prompt engineering. We then analyzed the effect of training on three dimensions: (1) the user experience of journalists when interacting with LLMs, (2) the accuracy of the texts (assessed by a domain expert), and (3) the reader perception, such as clarity, engagement, and other text quality dimensions (assessed by non-expert readers). Our results show: (1) Our training improved the perceived expertise of journalists but also decreased the perceived helpfulness of LLM use. (2) The effect on accuracy varied by the difficulty of the task. (3) There is a mixed impact of training on reader perception across different text quality dimensions.