Harnessing AI for efficient analysis of complex policy documents: a case study of Executive Order 14110

TL;DR

This paper evaluates AI's ability to analyze complex policy documents, demonstrating that certain AI systems can match human accuracy and efficiency, but reproducibility remains a challenge.

Contribution

It provides a case study assessing AI tools for policy analysis, highlighting their strengths and limitations compared to human experts.

Findings

AI systems Gemini 1.5 Pro and Claude 3 Opus performed comparably to humans.

AI analysis was significantly more efficient than manual review.

Reproducibility of AI results needs further improvement.

Abstract

Policy documents, such as legislation, regulations, and executive orders, are crucial in shaping society. However, their length and complexity make interpretation and application challenging and time-consuming. Artificial intelligence (AI), particularly large language models (LLMs), has the potential to automate the process of analyzing these documents, improving accuracy and efficiency. This study aims to evaluate the potential of AI in streamlining policy analysis and to identify the strengths and limitations of current AI approaches. The research focuses on question answering and tasks involving content extraction from policy documents. A case study was conducted using Executive Order 14110 on "Safe, Secure, and Trustworthy Development and Use of Artificial Intelligence" as a test case. Four commercial AI systems were used to analyze the document and answer a set of representative policy questions. The performance of the AI systems was compared to manual analysis conducted by human experts. The study found that two AI systems, Gemini 1.5 Pro and Claude 3 Opus, demonstrated significant potential for supporting policy analysis, providing accurate and reliable information extraction from complex documents. They performed comparably to human analysts but with significantly higher efficiency. However, achieving reproducibility remains a challenge, necessitating further research and development.