Software Doping Analysis for Human Oversight

TL;DR

This paper presents a framework combining formal analysis and probabilistic techniques to detect software doping and unfairness in high-risk decision systems, supporting human oversight and compliance with regulations.

Contribution

It introduces a novel black-box analysis method for identifying undesired software effects, applicable to emission systems and human evaluation systems, enhancing oversight capabilities.

Findings

Effective detection of software doping in emission systems

Identification of unfairness in high-risk human evaluation systems

Supports human-in-the-loop decision making

Abstract

This article introduces a framework that is meant to assist in mitigating societal risks that software can pose. Concretely, this encompasses facets of software doping as well as unfairness and discrimination in high-risk decision-making systems. The term software doping refers to software that contains surreptitiously added functionality that is against the interest of the user. A prominent example of software doping are the tampered emission cleaning systems that were found in millions of cars around the world when the diesel emissions scandal surfaced. The first part of this article combines the formal foundations of software doping analysis with established probabilistic falsification techniques to arrive at a black-box analysis technique for identifying undesired effects of software. We apply this technique to emission cleaning systems in diesel cars but also to high-risk systems that evaluate humans in a possibly unfair or discriminating way. We demonstrate how our approach can assist humans-in-the-loop to make better informed and more responsible decisions. This is to promote effective human oversight, which will be a central requirement enforced by the European Union's upcoming AI Act. We complement our technical contribution with a juridically, philosophically, and psychologically informed perspective on the potential problems caused by such systems.