Robustness of Algorithms for Causal Structure Learning to Hyperparameter Choice

TL;DR

This paper investigates how hyperparameter choices impact the performance of causal structure learning algorithms, emphasizing the importance of hyperparameter tuning for achieving optimal results in different data complexities.

Contribution

It provides an empirical analysis of hyperparameter effects on causal discovery algorithms, highlighting their influence on algorithm selection and performance.

Findings

Hyperparameters significantly influence causal structure learning outcomes.

Poor hyperparameter choices can lead to suboptimal algorithm selection.

Hyperparameter tuning is crucial for ensemble methods in causal discovery.

Abstract

Hyperparameters play a critical role in machine learning. Hyperparameter tuning can make the difference between state-of-the-art and poor prediction performance for any algorithm, but it is particularly challenging for structure learning due to its unsupervised nature. As a result, hyperparameter tuning is often neglected in favour of using the default values provided by a particular implementation of an algorithm. While there have been numerous studies on performance evaluation of causal discovery algorithms, how hyperparameters affect individual algorithms, as well as the choice of the best algorithm for a specific problem, has not been studied in depth before. This work addresses this gap by investigating the influence of hyperparameters on causal structure learning tasks. Specifically, we perform an empirical evaluation of hyperparameter selection for some seminal learning algorithms on datasets of varying levels of complexity. We find that, while the choice of algorithm remains crucial to obtaining state-of-the-art performance, hyperparameter selection in ensemble settings strongly influences the choice of algorithm, in that a poor choice of hyperparameters can lead to analysts using algorithms which do not give state-of-the-art performance for their data.