ACCBench: A Framework for Comparing Causality Algorithms

TL;DR

This paper introduces ACCBench, a benchmarking framework for comparing causality algorithms in socio-technical systems, evaluating their effectiveness, performance, and practical considerations through a detailed case study.

Contribution

It presents ACCBench, a tool for consistent evaluation of causality algorithms, and provides implementations and a case study analyzing their strengths and weaknesses.

Findings

Algorithms vary in effectiveness and performance.

Qualitative factors like human effort are important.

Case study highlights strengths and weaknesses of each algorithm.

Abstract

Modern socio-technical systems are increasingly complex. A fundamental problem is that the borders of such systems are often not well-defined a-priori, which among other problems can lead to unwanted behavior during runtime. Ideally, unwanted behavior should be prevented. If this is not possible the system shall at least be able to help determine potential cause(s) a-posterori, identify responsible parties and make them accountable for their behavior. Recently, several algorithms addressing these concepts have been proposed. However, the applicability of the corresponding approaches, specifically their effectiveness and performance, is mostly unknown. Therefore, in this paper, we propose ACCBench, a benchmark tool that allows to compare and evaluate causality algorithms under a consistent setting. Furthermore, we contribute an implementation of the two causality algorithms by G\"o{\ss}ler and Metayer and G\"o{\ss}ler and Astefanoaei as well as of a policy compliance approach based on some concepts of Main et al. Lastly, we conduct a case study of an Intelligent Door Control System, which exposes concrete strengths and weaknesses of all algorithms under different aspects. In the course of this, we show that the effectiveness of the algorithms in terms of cause detection as well as their performance differ to some extent. In addition, our analysis reports on some qualitative aspects that should be considered when evaluating each algorithm. For example, the human effort needed to configure the algorithm and model the use case is analyzed.