Toward domain generalized pruning by scoring out-of-distribution importance

TL;DR

This paper proposes a domain-aware filter pruning method that scores filter importance based on domain risk variance, improving cross-domain generalization of pruned neural networks.

Contribution

It introduces a novel importance scoring method considering out-of-distribution risks, enhancing domain generalization in filter pruning.

Findings

Improved cross-domain performance with the proposed pruning method.

Significant reduction in domain performance decay compared to baseline.

Demonstrated effectiveness across multiple domain shifts.

Abstract

Filter pruning has been widely used for compressing convolutional neural networks to reduce computation costs during the deployment stage. Recent studies have shown that filter pruning techniques can achieve lossless compression of deep neural networks, reducing redundant filters (kernels) without sacrificing accuracy performance. However, the evaluation is done when the training and testing data are from similar environmental conditions (independent and identically distributed), and how the filter pruning techniques would affect the cross-domain generalization (out-of-distribution) performance is largely ignored. We conduct extensive empirical experiments and reveal that although the intra-domain performance could be maintained after filter pruning, the cross-domain performance will decay to a large extent. As scoring a filter's importance is one of the central problems for pruning, we design the importance scoring estimation by using the variance of domain-level risks to consider the pruning risk in the unseen distribution. As such, we can remain more domain generalized filters. The experiments show that under the same pruning ratio, our method can achieve significantly better cross-domain generalization performance than the baseline filter pruning method. For the first attempt, our work sheds light on the joint problem of domain generalization and filter pruning research.