Provocation on Expertise in Social Impact Evaluations of Generative AI (and Beyond)

TL;DR

This paper critically examines the role of diverse expertise in conducting social impact evaluations of generative AI, emphasizing the need for integrating insights from various expert types.

Contribution

It introduces a framework for understanding the types of expertise necessary for robust social impact evaluations of generative AI and raises open questions for future research.

Findings

Highlighting the importance of diverse expertise in evaluations

Proposing a framework for integrating different expert insights

Identifying open questions for future work

Abstract

Social impact evaluations are emerging as a useful tool to understand, document, and evaluate the societal impacts of generative AI. In this provocation, we begin to think carefully about the types of experts and expertise that are needed to conduct robust social impact evaluations of generative AI. We suggest that doing so will require thoughtfully eliciting and integrating insights from a range of "domain experts" and "experiential experts," and close with five open questions.