AI Risk Profiles: A Standards Proposal for Pre-Deployment AI Risk Disclosures

TL;DR

This paper proposes a standardized risk profiling framework for AI systems to enhance transparency, guide deployment decisions, and support regulatory efforts, based on a comprehensive taxonomy and data collection methodology.

Contribution

It introduces a novel risk profiling standard for AI, including a taxonomy, data sourcing strategies, and a template-based approach for creating informative risk disclosures.

Findings

Applied the risk profiling methodology to prominent AI systems.

Demonstrated the utility of standardized profiles for risk assessment.

Discussed design considerations and future directions for the standard.

Abstract

As AI systems' sophistication and proliferation have increased, awareness of the risks has grown proportionally (Sorkin et al. 2023). In response, calls have grown for stronger emphasis on disclosure and transparency in the AI industry (NTIA 2023; OpenAI 2023b), with proposals ranging from standardizing use of technical disclosures, like model cards (Mitchell et al. 2019), to yet-unspecified licensing regimes (Sindhu 2023). Since the AI value chain is complicated, with actors representing various expertise, perspectives, and values, it is crucial that consumers of a transparency disclosure be able to understand the risks of the AI system the disclosure concerns. In this paper we propose a risk profiling standard which can guide downstream decision-making, including triaging further risk assessment, informing procurement and deployment, and directing regulatory frameworks. The standard is built on our proposed taxonomy of AI risks, which reflects a high-level categorization of the wide variety of risks proposed in the literature. We outline the myriad data sources needed to construct informative Risk Profiles and propose a template-based methodology for collating risk information into a standard, yet flexible, structure. We apply this methodology to a number of prominent AI systems using publicly available information. To conclude, we discuss design decisions for the profiles and future work.