P-MIA: A Profiled-Based Membership Inference Attack on Cognitive Diagnosis Models

TL;DR

This paper introduces P-MIA, a novel membership inference attack targeting cognitive diagnosis models, exploiting explainability visualizations to reveal sensitive student data, and demonstrates its effectiveness through extensive experiments.

Contribution

It is the first to systematically study MIA on CDMs, proposing a grey-box attack leveraging internal knowledge vectors exposed via visualizations.

Findings

P-MIA significantly outperforms black-box baselines.

The attack effectively reconstructs internal knowledge vectors.

P-MIA can evaluate machine unlearning techniques.

Abstract

Cognitive diagnosis models (CDMs) are pivotal for creating fine-grained learner profiles in modern intelligent education platforms. However, these models are trained on sensitive student data, raising significant privacy concerns. While membership inference attacks (MIA) have been studied in various domains, their application to CDMs remains a critical research gap, leaving their privacy risks unquantified. This paper is the first to systematically investigate MIA against CDMs. We introduce a novel and realistic grey box threat model that exploits the explainability features of these platforms, where a model's internal knowledge state vectors are exposed to users through visualizations such as radar charts. We demonstrate that these vectors can be accurately reverse-engineered from such visualizations, creating a potent attack surface. Based on this threat model, we propose a profile-based MIA (P-MIA) framework that leverages both the model's final prediction probabilities and the exposed internal knowledge state vectors as features. Extensive experiments on three real-world datasets against mainstream CDMs show that our grey-box attack significantly outperforms standard black-box baselines. Furthermore, we showcase the utility of P-MIA as an auditing tool by successfully evaluating the efficacy of machine unlearning techniques and revealing their limitations.