Assessing Generative AI value in a public sector context: evidence from a field experiment

TL;DR

This study evaluates the impact of Generative AI on complex knowledge tasks in the public sector through a field experiment, revealing mixed results depending on task type and highlighting practical insights.

Contribution

It provides empirical evidence on how Generative AI affects performance on real-world public sector tasks, with detailed analysis of task-specific outcomes.

Findings

Gen AI improved document understanding task performance by 17% in answer quality.

Gen AI reduced data analysis task quality by 12%, with no time difference.

Results indicate task-dependent benefits of Generative AI in public sector applications.

Abstract

The emergence of Generative AI (Gen AI) has motivated an interest in understanding how it could be used to enhance productivity across various tasks. We add to research results for the performance impact of Gen AI on complex knowledge-based tasks in a public sector setting. In a pre-registered experiment, after establishing a baseline level of performance, we find mixed evidence for two types of composite tasks related to document understanding and data analysis. For the Documents task, the treatment group using Gen AI had a 17% improvement in answer quality scores (as judged by human evaluators) and a 34% improvement in task completion time compared to a control group. For the Data task, we find the Gen AI treatment group experienced a 12% reduction in quality scores and no significant difference in mean completion time compared to the control group. These results suggest that the benefits of Gen AI may be task and potentially respondent dependent. We also discuss field notes and lessons learned, as well as supplementary insights from a post-trial survey and feedback workshop with participants.