Scoring the European Citizen in the AI Era

TL;DR

The paper examines the European AI Act's ban on social scoring, arguing it is a flexible tool designed to protect individuals from disproportionate AI-based scoring practices, despite existing in a context of permitted high-risk AI systems.

Contribution

It analyzes the legal and practical implications of the AI Act's social scoring ban, highlighting its potential as a safeguard against disproportionate AI scoring practices in Europe.

Findings

The ban is drafted flexibly to serve as a protective tool.

Existing scoring practices are classified as high-risk AI systems.

The ban complements GDPR protections against disproportionate AI use.

Abstract

Social scoring is one of the AI practices banned by the AI Act. This ban is explicitly inspired by China, which in 2014 announced its intention to set up a large-scale government project - the Social Credit System - aiming to rate every Chinese citizen according to their good behaviour, using digital technologies and AI. But in Europe, individuals are also scored by public and private bodies in a variety of contexts, such as assessing creditworthiness, monitoring employee productivity, detecting social fraud or terrorist risks, and so on. However, the AI Act does not intend to prohibit these types of scoring, as they would qualify as 'high-risk AI systems', which are authorised while subject to various requirements. One might therefore think that the ban on social scoring will have no practical effect on the scoring practices already in use in Europe, and that it is merely a vague safeguard in case an authoritarian power is tempted to set up such a system on European territory. Contrary to this view, this article argues that the ban has been drafted in a way that is flexible and therefore likely to make it a useful tool, similar and complementary to Article 22 of the General Data Protection Regulation, to protect individuals against certain forms of disproportionate use of AI-based scoring.