Conditional entropy minimization principle for learning domain invariant representation features

TL;DR

This paper introduces a conditional entropy minimization framework to improve domain invariant feature learning, effectively filtering out spurious features and enhancing generalization in domain generalization tasks.

Contribution

It proposes a novel CEM-based method that better isolates true invariant features, closely relates to the Information Bottleneck framework, and demonstrates improved generalization.

Findings

Achieves competitive accuracy on several DG datasets

Effectively filters out spurious invariant features

Theoretically recovers true invariant features under certain conditions

Abstract

Invariance-principle-based methods such as Invariant Risk Minimization (IRM), have recently emerged as promising approaches for Domain Generalization (DG). Despite promising theory, such approaches fail in common classification tasks due to the mixing of true invariant features and spurious invariant features. To address this, we propose a framework based on the conditional entropy minimization (CEM) principle to filter-out the spurious invariant features leading to a new algorithm with a better generalization capability. We show that our proposed approach is closely related to the well-known Information Bottleneck (IB) framework and prove that under certain assumptions, entropy minimization can exactly recover the true invariant features. Our approach provides competitive classification accuracy compared to recent theoretically-principled state-of-the-art alternatives across several DG datasets.