Structured Neural Networks for Density Estimation and Causal Inference

TL;DR

This paper introduces Structured Neural Networks (StrNN) that incorporate known invariances and independence structures into neural network models, improving density estimation and causal inference capabilities.

Contribution

It proposes a novel method to embed structure via masking in neural networks, linking architecture design to binary matrix factorization, and demonstrates applications in density estimation and causal inference.

Findings

StrNN effectively encodes independence structures in neural networks.

Structured Autoregressive Flows improve density estimation accuracy.

StrAFs enable causal effect estimation through interventional and counterfactual analysis.

Abstract

Injecting structure into neural networks enables learning functions that satisfy invariances with respect to subsets of inputs. For instance, when learning generative models using neural networks, it is advantageous to encode the conditional independence structure of observed variables, often in the form of Bayesian networks. We propose the Structured Neural Network (StrNN), which injects structure through masking pathways in a neural network. The masks are designed via a novel relationship we explore between neural network architectures and binary matrix factorization, to ensure that the desired independencies are respected. We devise and study practical algorithms for this otherwise NP-hard design problem based on novel objectives that control the model architecture. We demonstrate the utility of StrNN in three applications: (1) binary and Gaussian density estimation with StrNN, (2) real-valued density estimation with Structured Autoregressive Flows (StrAFs) and Structured Continuous Normalizing Flows (StrCNF), and (3) interventional and counterfactual analysis with StrAFs for causal inference. Our work opens up new avenues for learning neural networks that enable data-efficient generative modeling and the use of normalizing flows for causal effect estimation.