When Machine Learning Gets Personal: Evaluating Prediction and Explanation

TL;DR

This paper introduces a framework to evaluate how personalization in machine learning models affects both prediction accuracy and explanation clarity, revealing that these impacts can diverge and emphasizing the need for joint assessment.

Contribution

It provides a unified framework to quantify personalization effects on prediction and explanation, including a hypothesis test and bounds on detectability, with practical insights for real-world datasets.

Findings

Personalization can improve or worsen explainability independently of prediction accuracy.

A finite-sample lower bound on hypothesis test error depends on dataset characteristics.

Some effects of personalization are fundamentally untestable due to dataset limitations.

Abstract

In high-stakes domains like healthcare, users often expect that sharing personal information with machine learning systems will yield tangible benefits, such as more accurate diagnoses and clearer explanations of contributing factors. However, the validity of this assumption remains largely unexplored. We propose a unified framework to quantify how personalizing a model influences both prediction and explanation. We show that its impacts on prediction and explanation can diverge: a model may become more or less explainable even when prediction is unchanged. For practical settings, we study a standard hypothesis test for detecting personalization effects on demographic groups. We derive a finite-sample lower bound on its probability of error as a function of group sizes, number of personal attributes, and desired benefit from personalization. This provides actionable insights, such as which dataset characteristics are necessary to test an effect, or the maximum effect that can be tested given a dataset. We apply our framework to real-world tabular datasets using feature-attribution methods, uncovering scenarios where effects are fundamentally untestable due to the dataset statistics. Our results highlight the need for joint evaluation of prediction and explanation in personalized models and the importance of designing models and datasets with sufficient information for such evaluation.