Dynamic Documentation for AI Systems

TL;DR

This paper advocates for dynamic documentation of AI systems, highlighting its importance for transparency, and critiques current standards by comparing them to environmental impact statements, proposing a new paradigm.

Contribution

It introduces the concept of dynamic documentation for AI, analyzes limitations of current protocols, and evaluates new proposals like Reward Reports for comprehensive system understanding.

Findings

Current AI documentation standards are inadequate for modern architectures.

Environmental Impact Statements offer valuable insights but have limitations when applied to AI.

Dynamic documentation can improve transparency and accountability in AI systems.

Abstract

AI documentation is a rapidly-growing channel for coordinating the design of AI technologies with policies for transparency and accessibility. Calls to standardize and enact documentation of algorithmic harms and impacts are now commonplace. However, documentation standards for AI remain inchoate, and fail to match the capabilities and social effects of increasingly impactful architectures such as Large Language Models (LLMs). In this paper, we show the limits of present documentation protocols, and argue for dynamic documentation as a new paradigm for understanding and evaluating AI systems. We first review canonical approaches to system documentation outside the context of AI, focusing on the complex history of Environmental Impact Statements (EISs). We next compare critical elements of the EIS framework to present challenges with algorithmic documentation, which have inherited the limitations of EISs without incorporating their strengths. These challenges are specifically illustrated through the growing popularity of Model Cards and two case studies of algorithmic impact assessment in China and Canada. Finally, we evaluate more recent proposals, including Reward Reports, as potential components of fully dynamic AI documentation protocols.