Frontier AI Auditing: Toward Rigorous Third-Party Assessment of Safety and Security Practices at Leading AI Companies

TL;DR

This paper advocates for rigorous third-party audits of frontier AI developers to verify safety and security claims, proposing a structured framework with assurance levels to enhance trust and accountability in AI systems.

Contribution

It introduces a comprehensive vision for frontier AI auditing, including assurance levels, standards, and a roadmap for implementation and ecosystem growth.

Findings

Proposes four AI Assurance Levels for auditing.

Recommends baseline and advanced standards for different AI developer tiers.

Outlines key steps to achieve effective and high-quality AI auditing.

Abstract

We outline a vision for frontier AI auditing, which we define as rigorous third-party verification of frontier AI developers' safety and security claims, and evaluation of their systems and practices against relevant standards, based on deep, secure access to non-public information. Frontier AI audits should not be limited to a company's publicly deployed products, but should instead consider the full range of organization-level safety and security risks, including internal deployment of AI systems, information security practices, and safety decision-making processes. We describe four AI Assurance Levels (AALs), the higher levels of which provide greater confidence in audit findings. We recommend AAL-1 as a baseline for frontier AI generally, and AAL-2 as a near-term goal for the most advanced subset of frontier AI developers. Achieving the vision we outline will require (1) ensuring high quality standards for frontier AI auditing, so it does not devolve into a checkbox exercise or lag behind changes in the industry; (2) growing the ecosystem of audit providers at a rapid pace without compromising quality; (3) accelerating adoption of frontier AI auditing by clarifying and strengthening incentives; and (4) achieving technical readiness for high AI Assurance Levels so they can be applied when needed.