Fragile Giants: Understanding the Susceptibility of Models to Subpopulation Attacks

TL;DR

This paper investigates how increasing model complexity heightens vulnerability to subpopulation poisoning attacks, revealing that overparameterized models are more susceptible and often fail to detect attacks on small, interpretable subgroups.

Contribution

The authors introduce a theoretical framework explaining the link between model overparameterization and susceptibility to subpopulation poisoning, supported by extensive empirical validation.

Findings

More complex models are more vulnerable to subpopulation attacks.

Attacks on small subgroups often go undetected.

Overparameterized models memorize and misclassify targeted subpopulations.

Abstract

As machine learning models become increasingly complex, concerns about their robustness and trustworthiness have become more pressing. A critical vulnerability of these models is data poisoning attacks, where adversaries deliberately alter training data to degrade model performance. One particularly stealthy form of these attacks is subpopulation poisoning, which targets distinct subgroups within a dataset while leaving overall performance largely intact. The ability of these attacks to generalize within subpopulations poses a significant risk in real-world settings, as they can be exploited to harm marginalized or underrepresented groups within the dataset. In this work, we investigate how model complexity influences susceptibility to subpopulation poisoning attacks. We introduce a theoretical framework that explains how overparameterized models, due to their large capacity, can inadvertently memorize and misclassify targeted subpopulations. To validate our theory, we conduct extensive experiments on large-scale image and text datasets using popular model architectures. Our results show a clear trend: models with more parameters are significantly more vulnerable to subpopulation poisoning. Moreover, we find that attacks on smaller, human-interpretable subgroups often go undetected by these models. These results highlight the need to develop defenses that specifically address subpopulation vulnerabilities.