Score-informed Neural Operator for Enhancing Ordering-based Causal Discovery

TL;DR

This paper introduces SciNO, a neural operator that improves Hessian diagonal approximation in score-based causal discovery, leading to more accurate and scalable causal ordering, and enhances LLM causal reasoning without extra tuning.

Contribution

The paper proposes SciNO, a novel neural operator for stable Hessian approximation in score-based causal discovery, and a probabilistic control algorithm to improve LLM causal reasoning.

Findings

SciNO reduces order divergence by 42.7% on synthetic graphs.

SciNO reduces order divergence by 31.5% on real-world datasets.

The method maintains memory efficiency and scalability.

Abstract

Ordering-based approaches to causal discovery identify topological orders of causal graphs, providing scalable alternatives to combinatorial search methods. Under the Additive Noise Model (ANM) assumption, recent causal ordering methods based on score matching require an accurate estimation of the Hessian diagonal of the log-densities. In this paper, we aim to improve the approximation of the Hessian diagonal of the log-densities, thereby enhancing the performance of ordering-based causal discovery algorithms. Existing approaches that rely on Stein gradient estimators are computationally expensive and memory-intensive, while diffusion-model-based methods remain unstable due to the second-order derivatives of score models. To alleviate these problems, we propose Score-informed Neural Operator (SciNO), a probabilistic generative model in smooth function spaces designed to stably approximate the Hessian diagonal and to preserve structural information during the score modeling. Empirical results show that SciNO reduces order divergence by 42.7% on synthetic graphs and by 31.5% on real-world datasets on average compared to DiffAN, while maintaining memory efficiency and scalability. Furthermore, we propose a probabilistic control algorithm for causal reasoning with autoregressive models that integrates SciNO's probability estimates with autoregressive model priors, enabling reliable data-driven causal ordering informed by semantic information. Consequently, the proposed method enhances causal reasoning abilities of LLMs without additional fine-tuning or prompt engineering.