Effect of covariate shift on multi-class classification of Fermi-LAT sources

TL;DR

This study investigates how covariate shift impacts the accuracy of multi-class machine learning classification of Fermi-LAT sources, revealing that it mainly affects performance metrics rather than probability estimates.

Contribution

First quantitative analysis of covariate shift effects on Fermi-LAT source classification, introducing sample weighting to account for distribution differences.

Findings

Covariate shift has minimal effect on predicted probabilities.

Performance metrics like precision and recall can decrease by 10-20% due to covariate shift.

Weighted sampling can mitigate the impact of covariate shift on performance estimates.

Abstract

Probabilistic classification of unassociated Fermi-LAT sources using machine learning methods has an implicit assumption that the distributions of associated and unassociated sources are the same as a function of source parameters, which is not the case for the Fermi-LAT catalogs. The problem of different distributions of training and testing (or target) datasets as a function of input features (covariates) is known as the covariate shift. In this paper, we, for the first time, quantitatively estimate the effect of the covariate shift on the multi-class classification of Fermi-LAT sources. We introduce sample weights proportional to the ratio of unassociated to associated source probability density functions so that associated sources in areas, which are densely populated with unassociated sources, have more weight than the sources in areas with few unassociated sources. We find that the covariate shift has relatively little effect on the predicted probabilities, i.e., the training can be performed either with weighted or with unweighted samples, which is generally expected for the covariate shift problems. The main effect of the covariate shift is on the estimated performance of the classification. Depending on the class, the covariate shift can lead up to 10 - 20% reduction in precision and recall compared to the estimates, where the covariate shift is not taken into account.