"I know it when I see it". Visualization and Intuitive Interpretability

TL;DR

This paper explores how visualization affects intuitive interpretability of machine learning models, highlighting its benefits and limitations, and discusses human biases introduced through semantic concepts and the importance of avoiding singular representations.

Contribution

It provides a nuanced analysis of visualization's role in interpretability, emphasizing the need to understand its dual effects and human biases in model interpretation.

Findings

Visualization enables intuitive understanding but also introduces biases.

Dimensionality reduction and regularization are necessary pre-interpretation steps.

Avoiding singular representations can improve interpretability.

Abstract

Most research on the interpretability of machine learning systems focuses on the development of a more rigorous notion of interpretability. I suggest that a better understanding of the deficiencies of the intuitive notion of interpretability is needed as well. I show that visualization enables but also impedes intuitive interpretability, as it presupposes two levels of technical pre-interpretation: dimensionality reduction and regularization. Furthermore, I argue that the use of positive concepts to emulate the distributed semantic structure of machine learning models introduces a significant human bias into the model. As a consequence, I suggest that, if intuitive interpretability is needed, singular representations of internal model states should be avoided.