Qualit\"atsma{\ss}e bin\"arer Klassifikationen im Bereich kriminalprognostischer Instrumente der vierten Generation

TL;DR

This thesis evaluates the effectiveness of binary classifiers in criminal risk assessment tools, showing that positive predictive value aligns better with judicial decisions than AUC, with significant deviations observed in real-world data.

Contribution

It introduces a comparison between AUC and PPV_k for evaluating criminal risk classifiers and demonstrates the potential for large deviations, highlighting implications for judicial decision-making.

Findings

PPV_k models judicial decisions better than AUC

Deviations between measures can reach up to 0.75

Real-world data shows a deviation of 0.48 between measures

Abstract

This master's thesis discusses an important issue regarding how algorithmic decision making (ADM) is used in crime forecasting. In America forecasting tools are widely used by judiciary systems for making decisions about risk offenders based on criminal justice for risk offenders. By making use of such tools, the judiciary relies on ADM in order to make error free judgement on offenders. For this purpose, one of the quality measures for machine learning techniques which is widly used, the $AUC$ (area under curve), is compared to and contrasted for results with the $PPV_k$ (positive predictive value). Keeping in view the criticality of judgement along with a high dependency on tools offering ADM, it is necessary to evaluate risk tools that aid in decision making based on algorithms. In this methodology, such an evaluation is conducted by implementing a common machine learning approach called binary classifier, as it determines the binary outcome of the underlying juristic question. This thesis showed that the $PPV_k$ (positive predictive value) technique models the decision of judges much better than the $AUC$. Therefore, this research has investigated whether there exists a classifier for which the $PPV_k$ deviates from $AUC$ by a large proportion. It could be shown that the deviation can rise up to 0.75. In order to test this deviation on an already in used Classifier, data from the fourth generation risk assement tool COMPAS was used. The result were were quite alarming as the two measures derivate from each other by 0.48. In this study, the risk assessment evaluation of the forecasting tools was successfully conducted, carefully reviewed and examined. Additionally, it is also discussed whether such systems used for the purpose of making decisions should be socially accepted or not.