Fairness Increases Adversarial Vulnerability

TL;DR

This paper investigates the tradeoff between fairness and robustness in deep learning models, revealing that increasing fairness can reduce adversarial robustness, and proposes a method to balance both.

Contribution

It identifies the inherent tradeoff between fairness and robustness, analyzes the underlying causes, and offers a simple approach to improve both simultaneously.

Findings

Fairness can decrease model robustness to adversarial attacks.

Distance to decision boundary explains the fairness-robustness tradeoff.

Proposed method improves fairness and robustness tradeoffs in vision models.

Abstract

The remarkable performance of deep learning models and their applications in consequential domains (e.g., facial recognition) introduces important challenges at the intersection of equity and security. Fairness and robustness are two desired notions often required in learning models. Fairness ensures that models do not disproportionately harm (or benefit) some groups over others, while robustness measures the models' resilience against small input perturbations. This paper shows the existence of a dichotomy between fairness and robustness, and analyzes when achieving fairness decreases the model robustness to adversarial samples. The reported analysis sheds light on the factors causing such contrasting behavior, suggesting that distance to the decision boundary across groups as a key explainer for this behavior. Extensive experiments on non-linear models and different architectures validate the theoretical findings in multiple vision domains. Finally, the paper proposes a simple, yet effective, solution to construct models achieving good tradeoffs between fairness and robustness.