Do AI Companies Make Good on Voluntary Commitments to the White House?

TL;DR

This paper evaluates how well major AI companies adhere to voluntary commitments made to the White House, revealing significant gaps in transparency and accountability, especially regarding model weight security.

Contribution

It develops a detailed rubric to score companies' compliance with White House commitments and highlights structural shortcomings in current AI governance practices.

Findings

OpenAI scores highest at 83% compliance.

Average company compliance is 53%.

Model weight security compliance is notably poor at 17%.

Abstract

Voluntary commitments are central to international AI governance, as demonstrated by recent voluntary guidelines from the White House to the G7, from Bletchley Park to Seoul. How do major AI companies make good on their commitments? We score companies based on their publicly disclosed behavior by developing a detailed rubric based on their eight voluntary commitments to the White House in 2023. We find significant heterogeneity: while the highest-scoring company (OpenAI) scores a 83% overall on our rubric, the average score across all companies is just 53%. The companies demonstrate systemically poor performance for their commitment to model weight security with an average score of 17%: 11 of the 16 companies receive 0% for this commitment. Our analysis highlights a clear structural shortcoming that future AI governance initiatives should correct: when companies make public commitments, they should proactively disclose how they meet their commitments to provide accountability, and these disclosures should be verifiable. To advance policymaking on corporate AI governance, we provide three directed recommendations that address underspecified commitments, the role of complex AI supply chains, and public transparency that could be applied towards AI governance initiatives worldwide.