Improving Baselines in the Wild

TL;DR

This paper provides insights into training strategies and domain correlations in the WILDS benchmark, highlighting the importance of metric-specific validation, hyper-parameter tuning, and domain-label relationships for improving model robustness.

Contribution

The study offers new empirical observations on dataset-specific validation, hyper-parameter sensitivity, and domain-label correlations in WILDS, informing future robustness research.

Findings

Separate cross-validation per metric is essential.

Weak validation-test correlation complicates model development.

Minor hyper-parameter tweaks significantly boost performance.

Abstract

We share our experience with the recently released WILDS benchmark, a collection of ten datasets dedicated to developing models and training strategies which are robust to domain shifts. Several experiments yield a couple of critical observations which we believe are of general interest for any future work on WILDS. Our study focuses on two datasets: iWildCam and FMoW. We show that (1) Conducting separate cross-validation for each evaluation metric is crucial for both datasets, (2) A weak correlation between validation and test performance might make model development difficult for iWildCam, (3) Minor changes in the training of hyper-parameters improve the baseline by a relatively large margin (mainly on FMoW), (4) There is a strong correlation between certain domains and certain target labels (mainly on iWildCam). To the best of our knowledge, no prior work on these datasets has reported these observations despite their obvious importance. Our code is public.