Debunking Generalization Error or: How I Learned to Stop Worrying and Love My Training Set

TL;DR

This paper investigates how well machine learning models trained on simulations can predict galaxy properties from spectra, focusing on understanding and estimating their generalization error when applied to real observational data.

Contribution

It introduces a framework to model the generalization error based on the distance between simulation and real data domains, aiding in assessing model reliability.

Findings

Proposes a method to estimate generalization error from simulation to real data.

Highlights the importance of domain distance in model performance.

Provides insights into the transferability of astrophysical machine learning models.

Abstract

We aim to determine some physical properties of distant galaxies (for example, stellar mass, star formation history, or chemical enrichment history) from their observed spectra, using supervised machine learning methods. We know that different astrophysical processes leave their imprint in various regions of the spectra with characteristic signatures. Unfortunately, identifying a training set for this problem is very hard, because labels are not readily available - we have no way of knowing the true history of how galaxies have formed. One possible approach to this problem is to train machine learning models on state-of-the-art cosmological simulations. However, when algorithms are trained on the simulations, it is unclear how well they will perform once applied to real data. In this paper, we attempt to model the generalization error as a function of an appropriate measure of distance between the source domain and the application domain. Our goal is to obtain a reliable estimate of how a model trained on simulations might behave on data.