INSURE: An Information Theory Inspired Disentanglement and Purification Model for Domain Generalization

TL;DR

The paper introduces INSURE, a novel model inspired by information theory that explicitly disentangles class-relevant and domain-specific features to improve domain generalization performance.

Contribution

It proposes an efficient disentanglement method using a learnable binary mask and introduces new loss functions to ensure the sufficiency and compactness of class-relevant features.

Findings

INSURE outperforms state-of-the-art methods on four DG benchmarks.

Disentangling class-relevant features benefits domain generalization.

The model effectively isolates class and domain information for better generalization.

Abstract

Domain Generalization (DG) aims to learn a generalizable model on the unseen target domain by only training on the multiple observed source domains. Although a variety of DG methods have focused on extracting domain-invariant features, the domain-specific class-relevant features have attracted attention and been argued to benefit generalization to the unseen target domain. To take into account the class-relevant domain-specific information, in this paper we propose an Information theory iNspired diSentanglement and pURification modEl (INSURE) to explicitly disentangle the latent features to obtain sufficient and compact (necessary) class-relevant feature for generalization to the unseen domain. Specifically, we first propose an information theory inspired loss function to ensure the disentangled class-relevant features contain sufficient class label information and the other disentangled auxiliary feature has sufficient domain information. We further propose a paired purification loss function to let the auxiliary feature discard all the class-relevant information and thus the class-relevant feature will contain sufficient and compact (necessary) class-relevant information. Moreover, instead of using multiple encoders, we propose to use a learnable binary mask as our disentangler to make the disentanglement more efficient and make the disentangled features complementary to each other. We conduct extensive experiments on four widely used DG benchmark datasets including PACS, OfficeHome, TerraIncognita, and DomainNet. The proposed INSURE outperforms the state-of-art methods. We also empirically show that domain-specific class-relevant features are beneficial for domain generalization.