PEER pressure: Model-to-Model Regularization for Single Source Domain Generalization

TL;DR

PEER introduces a model-to-model regularization technique that stabilizes training and improves single source domain generalization by using a proxy model to accumulate knowledge and reduce feature distortion.

Contribution

The paper proposes PEER, a novel regularization method employing a proxy model and mutual information maximization to enhance domain generalization and reduce performance fluctuation.

Findings

PEER reduces out-of-distribution performance fluctuation.

PEER achieves state-of-the-art results with simple augmentation.

PEER outperforms prior methods on multiple datasets.

Abstract

Data augmentation is a popular tool for single source domain generalization, which expands the source domain by generating simulated ones, improving generalization on unseen target domains. In this work, we show that the performance of such augmentation-based methods in the target domains universally fluctuates during training, posing challenges in model selection under realistic scenarios. We argue that the fluctuation stems from the inability of the model to accumulate the knowledge learned from diverse augmentations, exacerbating feature distortion during training. Based on this observation, we propose a novel generalization method, coined Parameter-Space Ensemble with Entropy Regularization (PEER), that uses a proxy model to learn the augmented data on behalf of the main model. The main model is updated by averaging its parameters with the proxy model, progressively accumulating knowledge over the training steps. Maximizing the mutual information between the output representations of the two models guides the learning process of the proxy model, mitigating feature distortion during training. Experimental results demonstrate the effectiveness of PEER in reducing the OOD performance fluctuation and enhancing generalization across various datasets, including PACS, Digits, Office-Home, and VLCS. Notably, our method with simple random augmentation achieves state-of-the-art performance, surpassing prior approaches on sDG that utilize complex data augmentation strategies.