Towards a Better Evaluation of Out-of-Domain Generalization

TL;DR

This paper critically examines the common average measure for evaluating out-of-domain generalization, proposes a more robust worst+gap measure supported by theory, and validates its effectiveness through extensive experiments on modified datasets.

Contribution

It introduces the worst+gap measure as a new evaluation metric for domain generalization, supported by theoretical analysis and empirical validation.

Findings

Average measure poorly approximates true DG performance.

Worst+gap measure is more robust and reliable.

Experimental results confirm the superiority of the proposed measure.

Abstract

The objective of Domain Generalization (DG) is to devise algorithms and models capable of achieving high performance on previously unseen test distributions. In the pursuit of this objective, average measure has been employed as the prevalent measure for evaluating models and comparing algorithms in the existing DG studies. Despite its significance, a comprehensive exploration of the average measure has been lacking and its suitability in approximating the true domain generalization performance has been questionable. In this study, we carefully investigate the limitations inherent in the average measure and propose worst+gap measure as a robust alternative. We establish theoretical grounds of the proposed measure by deriving two theorems starting from two different assumptions. We conduct extensive experimental investigations to compare the proposed worst+gap measure with the conventional average measure. Given the indispensable need to access the true DG performance for studying measures, we modify five existing datasets to come up with SR-CMNIST, C-Cats&Dogs, L-CIFAR10, PACS-corrupted, and VLCS-corrupted datasets. The experiment results unveil an inferior performance of the average measure in approximating the true DG performance and confirm the robustness of the theoretically supported worst+gap measure.