Training Compute Thresholds: Features and Functions in AI Regulation

TL;DR

This paper advocates for using training compute thresholds as a primary metric for AI regulation, highlighting their advantages in identifying potentially risky GPAI models early and reliably, while emphasizing the need for supplementary measures.

Contribution

It argues that training compute is the most suitable initial regulatory metric for GPAI, due to its correlation with capabilities, verifiability, and early measurability, and discusses how thresholds should be integrated into oversight.

Findings

Training compute correlates with model capabilities and risks.

Compute thresholds can be measured early and verified externally.

Thresholds should trigger further evaluation and not be sole risk determinants.

Abstract

Regulators in the US and EU are using thresholds based on training compute--the number of computational operations used in training--to identify general-purpose artificial intelligence (GPAI) models that may pose risks of large-scale societal harm. We argue that training compute currently is the most suitable metric to identify GPAI models that deserve regulatory oversight and further scrutiny. Training compute correlates with model capabilities and risks, is quantifiable, can be measured early in the AI lifecycle, and can be verified by external actors, among other advantageous features. These features make compute thresholds considerably more suitable than other proposed metrics to serve as an initial filter to trigger additional regulatory requirements and scrutiny. However, training compute is an imperfect proxy for risk. As such, compute thresholds should not be used in isolation to determine appropriate mitigation measures. Instead, they should be used to detect potentially risky GPAI models that warrant regulatory oversight, such as through notification requirements, and further scrutiny, such as via model evaluations and risk assessments, the results of which may inform which mitigation measures are appropriate. In fact, this appears largely consistent with how compute thresholds are used today. As GPAI technology and market structures evolve, regulators should update compute thresholds and complement them with other metrics into regulatory review processes.