Practical Causal Evaluation Metrics for Biological Networks

TL;DR

This paper introduces the sign-augmented Structural Intervention Distance (sSID), a new metric for evaluating inferred biological causal networks that accounts for qualitative intervention effects, improving biological relevance.

Contribution

The paper develops the sSID metric tailored for biological networks, addressing the limitations of existing quantitative metrics by incorporating qualitative intervention effects.

Findings

sSID identifies different optimal algorithms compared to traditional metrics

Networks evaluated with sSID perform better in clinical covariate classification

sSID distinguishes structurally correct but functionally incorrect networks

Abstract

Estimating causal networks from biological data is a critical step in systems biology. When evaluating the inferred network, assessing the networks based on their intervention effects is particularly important for downstream probabilistic reasoning and the identification of potential drug targets. In the context of gene regulatory network inference, biological databases are often used as reference sources. These databases typically describe relationships in a qualitative rather than quantitative manner. However, few evaluation metrics have been developed that take this qualitative nature into account. To address this, we developed a metric, the sign-augmented Structural Intervention Distance (sSID), and a weighted sSID that incorporates the net effects of the intervention. Through simulations and analyses of real transcriptomic datasets, we found that our proposed metrics could identify a different algorithm as optimal compared to conventional metrics, and the network selected by sSID had a superior performance in the classification task of clinical covariates using transcriptomic data. This suggests that sSID can distinguish networks that are structurally correct but functionally incorrect, highlighting its potential as a more biologically meaningful and practical evaluation metric.