Machine Learning and Knowledge: Why Robustness Matters

TL;DR

This paper emphasizes that trust in machine learning should be based on the models' ability to provide knowledge through robustness and correct reasoning, not just reliability, highlighting the importance of interpretability and distribution shift robustness.

Contribution

It introduces an epistemic perspective on trust in machine learning, linking knowledge to model robustness and reasoning, and explains why properties like interpretability are crucial beyond reliability.

Findings

Robustness is essential for models to be considered as providing knowledge.

Interpretability and causal independence are important for trust beyond mere reliability.

Models must perform well across counterfactual scenarios to be trustworthy.

Abstract

Trusting machine learning algorithms requires having confidence in their outputs. Confidence is typically interpreted in terms of model reliability, where a model is reliable if it produces a high proportion of correct outputs. However, model reliability does not address concerns about the robustness of machine learning models, such as models relying on the wrong features or variations in performance based on context. I argue that the epistemic dimension of trust can instead be understood through the concept of knowledge, where the trustworthiness of an algorithm depends on whether its users are in the position to know that its outputs are correct. Knowledge requires beliefs to be formed for the right reasons and to be robust to error, so machine learning algorithms can only provide knowledge if they work well across counterfactual scenarios and if they make decisions based on the right features. This, I argue, can explain why we should care about model properties like interpretability, causal shortcut independence, and distribution shift robustness even if such properties are not required for model reliability.