Pruning has a disparate impact on model accuracy

TL;DR

This paper investigates how network pruning, a common model compression method, can lead to or worsen disparities in model accuracy across different groups, and proposes solutions to mitigate this issue.

Contribution

It identifies key factors like gradient norms and decision boundary distances that cause disparities, offering both theoretical analysis and practical mitigation strategies.

Findings

Pruning can create or worsen accuracy disparities across groups.

Differences in gradient norms and decision boundary distances are responsible for disparities.

Proposed mitigation reduces disparities caused by pruning.

Abstract

Network pruning is a widely-used compression technique that is able to significantly scale down overparameterized models with minimal loss of accuracy. This paper shows that pruning may create or exacerbate disparate impacts. The paper sheds light on the factors to cause such disparities, suggesting differences in gradient norms and distance to decision boundary across groups to be responsible for this critical issue. It analyzes these factors in detail, providing both theoretical and empirical support, and proposes a simple, yet effective, solution that mitigates the disparate impacts caused by pruning.