Domain Generalization Guided by Gradient Signal to Noise Ratio of Parameters

TL;DR

This paper introduces a novel domain generalization method that uses gradient signal-to-noise ratio to guide parameter dropout, combined with meta-learning to optimize dropout ratios, improving performance under domain shift.

Contribution

The paper proposes a GSNR-based dropout technique with meta-learning for domain generalization, moving beyond classical Bernoulli dropout methods.

Findings

Achieves competitive results on standard domain generalization benchmarks.

Improves robustness in classification and face anti-spoofing tasks.

Demonstrates effectiveness of GSNR-guided regularization under domain shift.

Abstract

Overfitting to the source domain is a common issue in gradient-based training of deep neural networks. To compensate for the over-parameterized models, numerous regularization techniques have been introduced such as those based on dropout. While these methods achieve significant improvements on classical benchmarks such as ImageNet, their performance diminishes with the introduction of domain shift in the test set i.e. when the unseen data comes from a significantly different distribution. In this paper, we move away from the classical approach of Bernoulli sampled dropout mask construction and propose to base the selection on gradient-signal-to-noise ratio (GSNR) of network's parameters. Specifically, at each training step, parameters with high GSNR will be discarded. Furthermore, we alleviate the burden of manually searching for the optimal dropout ratio by leveraging a meta-learning approach. We evaluate our method on standard domain generalization benchmarks and achieve competitive results on classification and face anti-spoofing problems.