Acceptable risks in Europe's proposed AI Act: Reasonableness and other principles for deciding how much risk management is enough

TL;DR

This paper critically examines the European AI Act's risk management approach, advocating for a reasonableness-based framework that balances trustworthiness, proportionality, and civic legitimacy in high-risk AI regulation.

Contribution

It proposes a more workable risk acceptability framework based on reasonableness, informed by legal principles and stakeholder input, improving the balance between regulation and trustworthiness.

Findings

The current 'as far as possible' criterion is unworkable and ineffective.

Reasonableness and cost-benefit analysis offer a more practical approach.

Stakeholder involvement enhances legitimacy and effectiveness of risk judgments.

Abstract

This paper critically evaluates the European Commission's proposed AI Act's approach to risk management and risk acceptability for high-risk AI systems that pose risks to fundamental rights and safety. The Act aims to promote "trustworthy" AI with a proportionate regulatory burden. Its provisions on risk acceptability require residual risks from high-risk systems to be reduced or eliminated "as far as possible", having regard to the "state of the art". This criterion, especially if interpreted narrowly, is unworkable and promotes neither proportionate regulatory burden, nor trustworthiness. By contrast the Parliament's most recent draft amendments to the risk management provisions introduce "reasonableness", cost-benefit analysis, and are more transparent about the value-laden and contextual nature of risk acceptability judgements. This paper argues that the Parliament's approach is more workable, and better balances the goals of proportionality and trustworthiness. It explains what reasonableness in risk acceptability judgments would entail, drawing on principles from negligence law and European medical devices regulation. And it contends that the approach to risk acceptability judgments need a firm foundation of civic legitimacy: including detailed guidance or involvement from regulators, and meaningful input from affected stakeholders.