Preserving Silent Features for Domain Generalization

TL;DR

This paper identifies that silent features, which are more generalizable, are suppressed during supervised fine-tuning, and proposes a method to preserve these features to enhance domain generalization performance.

Contribution

The paper introduces the concept of silent features, models their suppression during fine-tuning, and proposes STEP, a method to preserve silent features for better domain generalization.

Findings

STEP achieves state-of-the-art results on DG benchmarks.

Preserving silent features improves robustness to distribution shifts.

Theoretical analysis supports the benefit of silent feature preservation.

Abstract

Domain generalization (DG) aims to improve the generalization ability of the model trained on several known training domains over unseen test domains. Previous work has shown that self-supervised contrastive pre-training improves the robustness of the model on downstream tasks. However, in this paper, we find that self-supervised models do not exhibit better generalization performance than supervised models pre-trained on the same dataset in the DG setting. We argue that this is owing to the fact that the richer intra-class discriminative features extracted by self-supervised contrastive learning, which we term silent features, are suppressed during supervised fine-tuning. These silent features are likely to contain features that are more generalizable on the test domain. In this work, we model and analyze this feature suppression phenomenon and theoretically prove that preserving silent features can achieve lower expected test domain risk under certain conditions. In light of this, we propose a simple yet effective method termed STEP (Silent Feature Preservation) to improve the generalization performance of the self-supervised contrastive learning pre-trained model by alleviating the suppression of silent features during the supervised fine-tuning process. Experimental results show that STEP exhibits state-of-the-art performance on standard DG benchmarks with significant distribution shifts.