Structure Learning with Continuous Optimization: A Sober Look and Beyond

TL;DR

This paper critically examines the effectiveness of continuous optimization methods for DAG structure learning, analyzing their limitations under different noise conditions and proposing directions for more reliable and comprehensive approaches.

Contribution

It challenges previous assumptions about the success of continuous methods, providing counterexamples and emphasizing the importance of non-equal noise variances in future research.

Findings

Continuous methods may not perform well after data standardization.

Nonconvexity is a significant challenge, especially with non-equal noise variances.

Thresholding and sparsity significantly influence the final structure solutions.

Abstract

This paper investigates in which cases continuous optimization for directed acyclic graph (DAG) structure learning can and cannot perform well and why this happens, and suggests possible directions to make the search procedure more reliable. Reisach et al. (2021) suggested that the remarkable performance of several continuous structure learning approaches is primarily driven by a high agreement between the order of increasing marginal variances and the topological order, and demonstrated that these approaches do not perform well after data standardization. We analyze this phenomenon for continuous approaches assuming equal and non-equal noise variances, and show that the statement may not hold in either case by providing counterexamples, justifications, and possible alternative explanations. We further demonstrate that nonconvexity may be a main concern especially for the non-equal noise variances formulation, while recent advances in continuous structure learning fail to achieve improvement in this case. Our findings suggest that future works should take into account the non-equal noise variances formulation to handle more general settings and for a more comprehensive empirical evaluation. Lastly, we provide insights into other aspects of the search procedure, including thresholding and sparsity, and show that they play an important role in the final solutions.