HMOE: Hypernetwork-based Mixture of Experts for Domain Generalization

TL;DR

HMOE is a novel hypernetwork-based mixture of experts model for domain generalization that does not require domain labels, offers interpretability, and achieves state-of-the-art results by effectively clustering data according to domain shifts.

Contribution

This paper introduces HMOE, a hypernetwork-based MoE approach that is label-free and interpretable, improving domain generalization performance.

Findings

HMOE effectively clusters data into meaningful domains aligned with human intuition.

HMOE achieves state-of-the-art accuracy on multiple datasets.

HMOE surpasses existing methods significantly in average accuracy.

Abstract

Due to domain shifts, machine learning systems typically struggle to generalize well to new domains that differ from those of training data, which is what domain generalization (DG) aims to address. Although a variety of DG methods have been proposed, most of them fall short in interpretability and require domain labels, which are not available in many real-world scenarios. This paper presents a novel DG method, called HMOE: Hypernetwork-based Mixture of Experts (MoE), which does not rely on domain labels and is more interpretable. MoE proves effective in identifying heterogeneous patterns in data. For the DG problem, heterogeneity arises exactly from domain shifts. HMOE employs hypernetworks taking vectors as input to generate the weights of experts, which promotes knowledge sharing among experts and enables the exploration of their similarities in a low-dimensional vector space. We benchmark HMOE against other DG methods under a fair evaluation framework -- DomainBed. Our extensive experiments show that HMOE can effectively separate mixed-domain data into distinct clusters that are surprisingly more consistent with human intuition than original domain labels. Using self-learned domain information, HMOE achieves state-of-the-art results on most datasets and significantly surpasses other DG methods in average accuracy across all datasets.