Robust Representation Learning with Self-Distillation for Domain Generalization

TL;DR

This paper introduces RRLD, a novel self-distillation method for vision transformers that enhances domain generalization by learning robust features invariant to domain shifts and augmentations, validated on multiple datasets.

Contribution

The paper proposes a new self-distillation approach with intermediate-block and augmentation-guided strategies for improved domain generalization in vision transformers.

Findings

Achieves 1.2% to 2.3% accuracy improvement over state-of-the-art.

Effective on multiple benchmark datasets including PACS, OfficeHome, and industrial data.

Enhances robustness and generalization to unseen domains.

Abstract

Despite the recent success of deep neural networks, there remains a need for effective methods to enhance domain generalization using vision transformers. In this paper, we propose a novel domain generalization technique called Robust Representation Learning with Self-Distillation (RRLD) comprising i) intermediate-block self-distillation and ii) augmentation-guided self-distillation to improve the generalization capabilities of transformer-based models on unseen domains. This approach enables the network to learn robust and general features that are invariant to different augmentations and domain shifts while effectively mitigating overfitting to source domains. To evaluate the effectiveness of our proposed method, we perform extensive experiments on PACS and OfficeHome benchmark datasets, as well as an industrial wafer semiconductor defect dataset. The results demonstrate that RRLD achieves robust and accurate generalization performance. We observe an average accuracy improvement in the range of 1.2% to 2.3% over the state-of-the-art on the three datasets.