Learning from the machine: interpreting machine learning algorithms for point- and extended- source classification

TL;DR

This paper explores interpretability techniques for machine learning models in star-galaxy classification, improving understanding and accuracy using decision boundaries, feature selection, and model visualization on SDSS data.

Contribution

It introduces methods for interpreting black-box ML models in astronomy, including decision boundary visualization, feature pre-selection, and model explanation tools, applied to star-galaxy classification.

Findings

Improved classification accuracy by up to 33% using complex decision boundaries.

Demonstrated effective feature importance analysis on an object-by-object basis.

Enhanced understanding of model decisions through visualization and interpretation tools.

Abstract

We investigate star-galaxy classification for astronomical surveys in the context of four methods enabling the interpretation of black-box machine learning systems. The first is outputting and exploring the decision boundaries as given by decision tree based methods, which enables the visualization of the classification categories. Secondly, we investigate how the Mutual Information based Transductive Feature Selection (MINT) algorithm can be used to perform feature pre-selection. If one would like to provide only a small number of input features to a machine learning classification algorithm, feature pre-selection provides a method to determine which of the many possible input properties should be selected. Third is the use of the tree-interpreter package to enable popular decision tree based ensemble methods to be opened, visualized, and understood. This is done by additional analysis of the tree based model, determining not only which features are important to the model, but how important a feature is for a particular classification given its value. Lastly, we use decision boundaries from the model to revise an already existing method of classification, essentially asking the tree based method where decision boundaries are best placed and defining a new classification method. We showcase these techniques by applying them to the problem of star-galaxy separation using data from the Sloan Digital Sky Survey (hereafter SDSS). We use the output of MINT and the ensemble methods to demonstrate how more complex decision boundaries improve star-galaxy classification accuracy over the standard SDSS frames approach (reducing misclassifications by up to $\approx33\%$). We then show how tree-interpreter can be used to explore how relevant each photometric feature is when making a classification on an object by object basis.