Measurement in AI Policy: Opportunities and Challenges

TL;DR

This paper reviews the challenges and opportunities in measuring AI systems and their societal impact, aiming to guide future research in this vital area.

Contribution

It provides a comprehensive survey of measurement problems and summarizes discussions from a workshop, highlighting key challenges and research directions.

Findings

Identified six core challenges in AI measurement

Summarized insights from over 40 presentations and discussions

Proposed research agendas for advancing AI measurement

Abstract

As artificial intelligence increasingly influences our world, it becomes crucial to assess its technical progress and societal impact. This paper surveys problems and opportunities in the measurement of AI systems and their impact, based on a workshop held at Stanford University in the fall of 2019. We identify six summary challenges inherent to measuring the progress and impact of AI, and summarize over 40 presentations and associated discussions from the workshop. We hope this can inspire research agendas in this crucial area.