A review of possible effects of cognitive biases on the interpretation of rule-based machine learning models

TL;DR

This paper reviews how cognitive biases from psychology may influence human interpretation of rule-based machine learning models, emphasizing the need for empirical studies to understand and mitigate these effects.

Contribution

It bridges cognitive psychology and machine learning by analyzing cognitive biases affecting interpretability and proposing debiasing techniques for model designers.

Findings

Identifies 20 cognitive biases relevant to model interpretation

Highlights the gap between psychological findings and machine learning practice

Suggests debiasing techniques for improving interpretability

Abstract

While the interpretability of machine learning models is often equated with their mere syntactic comprehensibility, we think that interpretability goes beyond that, and that human interpretability should also be investigated from the point of view of cognitive science. The goal of this paper is to discuss to what extent cognitive biases may affect human understanding of interpretable machine learning models, in particular of logical rules discovered from data. Twenty cognitive biases are covered, as are possible debiasing techniques that can be adopted by designers of machine learning algorithms and software. Our review transfers results obtained in cognitive psychology to the domain of machine learning, aiming to bridge the current gap between these two areas. It needs to be followed by empirical studies specifically focused on the machine learning domain.