Learning discrete Bayesian networks in polynomial time and sample complexity

TL;DR

This paper presents a polynomial-time, sample-efficient method for learning the structure of discrete Bayesian networks under certain conditions, using group l12-regularized regression without assuming specific conditional distributions.

Contribution

It introduces a novel approach that guarantees exact structure recovery with polynomial time and logarithmic sample complexity for bounded-degree networks, under certain technical conditions.

Findings

Exact structure recovery under technical conditions

Polynomial time algorithm for Bayesian network learning

Logarithmic sample complexity for bounded-degree networks

Abstract

In this paper, we study the problem of structure learning for Bayesian networks in which nodes take discrete values. The problem is NP-hard in general but we show that under certain conditions we can recover the true structure of a Bayesian network with sufficient number of samples. We develop a mathematical model which does not assume any specific conditional probability distributions for the nodes. We use a primal-dual witness construction to prove that, under some technical conditions on the interaction between node pairs, we can do exact recovery of the parents and children of a node by performing group l_12-regularized multivariate regression. Thus, we recover the true Bayesian network structure. If degree of a node is bounded then the sample complexity of our proposed approach grows logarithmically with respect to the number of nodes in the Bayesian network. Furthermore, our method runs in polynomial time.

Tables

Table 1. Table 1 : F 1 subscript 𝐹 1 F_{1} -Scores and standard errors at 95 % percent 95 95\% confidence level on benchmark Bayesian networks. First two columns compare skeletons. Last six columns compare DAGs.

	${𝑭}_{\mathrm{𝟏}}$-Score
Network	Our	MMPC	Our Method	MMHC	Greedy	Sparse	Optimal	Exact
($𝒏=\left|𝑽\right|$)	Method		+ Greedy		Search	Candidate	Reinsertion	LP
alarm (37)	$0.74\pm 0.01$	$0.90\pm 0.01$	$0.92\pm 0.02$	$0.93\pm 0.02$	$0.87\pm 0.07$	$0.54\pm 0.06$	$0.86\pm 0.04$	$0.82\pm 0.02$
andes (223)	$0.77\pm 0.01$	$0.79\pm 0.01$	$0.82\pm 0.01$	$0.88\pm 0.01$	$0.79\pm 0.02$	$0.05\pm 0.02$	$0.74\pm 0.01$	$0.67\pm 0.01$
barley (48)	$0.37\pm 0.01$	$0.60\pm 0.01$	$0.74\pm 0.06$	$0.73\pm 0.05$	$0.61\pm 0.05$	$0.61\pm 0.04$	$0.67\pm 0.1$	$0.77\pm 0.01$
carpo (60)	$0.74\pm 0.01$	$0.78\pm 0.01$	$0.89\pm 0.01$	$0.86\pm 0.03$	$0.78\pm 0.02$	$0.15\pm 0.04$	$0.82\pm 0.02$	$0.69\pm 0.02$
child (20)	$0.94\pm 0.01$	$0.93\pm 0.01$	$0.98\pm 0.00$	$1\pm 0.00$	$1\pm 0.00$	$0.79\pm 0.11$	$0.96\pm 0.05$	$1\pm 0.00$
hailfinder (56)	$0.57\pm 0.02$	$0.58\pm 0.01$	$0.65\pm 0.02$	$0.75\pm 0.14$	$0.68\pm 0.15$	$0.46\pm 0.08$	$0.71\pm 0.1$	$0.68\pm 0.04$
mildew (35)	$0.30\pm 0.01$	$0.45\pm 0.02$	$0.83\pm 0.01$	$0.77\pm 0.00$	$0.70\pm 0.08$	$0.68\pm 0.04$	$0.65\pm 0.08$	$0.72\pm 0.00$
water (32)	$0.59\pm 0.02$	$0.63\pm 0.01$	$0.61\pm 0.02$	$0.60\pm 0.05$	$0.51\pm 0.06$	$0.28\pm 0.03$	$0.62\pm 0.06$	$0.61\pm 0.03$
win95pts (76)	$0.67\pm 0.01$	$0.77\pm 0.03$	$0.76\pm 0.01$	$0.79\pm 0.03$	$0.62\pm 0.02$	$0.13\pm 0.06$	$0.62\pm 0.04$	$0.59\pm 0.02$

Table 2. Table 2 : Negative log-likelihood and standard errors at 95 % percent 95 95\% confidence level on real world datasets.

	Negative Log-likelihood
Network	Our Method	MMHC	Greedy
($n=\left|V\right|$)	+ Greedy
dna (180)	$80.37\pm 0.24$	$80.57\pm 0.24$	$80.45\pm 0.25$
moviereview (1001)	$339.27\pm 8.85$	$340.60\pm 9.03$	$343.64\pm 9.28$
retail (135)	$10.88\pm 0.21$	$10.88\pm 0.21$	$10.88\pm 0.21$
audio(125)	$40.50\pm 0.57$	$40.28\pm 0.57$	$40.27\pm 0.57$
autos (26)	$18.61\pm 2.98$	$24.45\pm 1.74$	$18.62\pm 3.27$
jester(100)	$53.78\pm 0.39$	$53.53\pm 0.40$	$53.53\pm 0.40$
netflix (100)	$57.02\pm 0.23$	$56.85\pm 0.23$	$56.85\pm 0.23$
r52 (889)	$90.46\pm 3.43$	$86.92\pm 3.46$	$87.30\pm 3.49$
student-por (33)	$32.22\pm 0.72$	$33.88\pm 0.69$	$32.08\pm 0.73$
tmovie(500)	$55.33\pm 4.63$	$54.59\pm 4.65$	$54.96\pm 4.75$
webkb (839)	$159.39\pm 6.34$	$156.89\pm 6.28$	$157.75\pm 6.46$
promoters (58)	$78.03\pm 1.84$	$79.01\pm 2.10$	$79.01\pm 2.10$
sponge(45)	$26.36\pm 4.64$	$29.50\pm 3.54$	$25.49\pm 4.65$
triazines(59)	$13.78\pm 1.75$	$13.89\pm 1.60$	$12.83\pm 2.10$
wiki4he(53)	$57.99\pm 1.29$	$58.08\pm 1.34$	$58.05\pm 1.34$

Table 3. Table 3 : Comparison of mutual incoherence assumption on different supports

Number	Max	Support : Parent and Children	Support : Markov Blanket	MIPC is
of nodes	Degree	(MIPC) holds	(MIMB) holds	weaker than MIMB
100	7	99.7%	97.6%	91.9%
500	7	99.76%	96.82%	93.26%
1000	7	99.62%	95.95%	92.97%

Equations

$$\mathbb{P}(\mathbb{X})=\prod _{r=1}^n\mathbb{P}(X_r|X_{{\mathbf{\pi }}_r})$$

$$\text{A}=\left[\begin{array}{c}{\text{A}}_1\\ ⋮\\ {\text{A}}_k\end{array}\right]\text{where each Ai∈pi×q.}$$

$${\text{W}}_{{\mathbf{S}}_r}^{\ast }={\mathbb{E}}_{\mathbb{X}}[\mathcal{E}(X_{\bar{r}}{)}_{{\mathbf{S}}_r}\mathcal{E}(X_{\bar{r}}{)}_{{\mathbf{S}}_r}^{⊺}{]}^{-1}{\mathbb{E}}_{\mathbb{X}}[\mathcal{E}(X_{\bar{r}}{)}_{{\mathbf{S}}_r}\mathcal{E}(X_r{)}^{⊺}]$$

$${\mathbb{E}}_{\mathbb{X}}[X_{{\mathbf{S}}_r}{X}_{{\mathbf{S}}_r}^{⊺}{]}^{-1}{\mathbb{E}}_{\mathbb{X}}[X_{{\mathbf{S}}_r}{X}_r]\in (-\{0\}{)}^{|{\mathbf{S}}_r|}.$$

$$\mathcal{E}(X_r)\triangleq \mathcal{F}(\mathcal{E}(X_{\bar{r}});{\text{W}}^{\ast }).$$

$$\mathcal{E}(X_r)={{\text{W}}^{\ast }}^{⊺}\mathcal{E}(X_{\bar{r}})+\text{e}$$

$${\text{W}}^{\ast }=\left[\begin{array}{c}{\text{W}}_1^{\ast }\\ ⋮\\ {\text{W}}_{r-1}^{\ast }\\ {\text{W}}_{r+1}^{\ast }\\ ⋮\\ {\text{W}}_n^{\ast }\end{array}\right]$$

$$\begin{array}{cccc}{\text{W}}^{\ast }& =& \arg {\min }_{\text{W}}& \frac{1}{2}{\mathbb{E}}_{\mathbb{X}}[\Vert \mathcal{E}(X_r)-{\text{W}}^{⊺}\mathcal{E}(X_{\bar{r}}){\Vert }_2^2]\\ & & \text{such that}& {\text{W}}_i=0,\forall i\in ({\mathbf{\pi }}_r\cup {\mathbf{c}}_r{)}^c\end{array}$$

$$\begin{array}{rl}{\text{W}}^{\ast }& =({\text{W}}_{{\mathbf{S}}_r.}^{\ast };0)\\ {\text{W}}_{{\mathbf{S}}_r.}^{\ast }& ={\mathbb{E}}_{\mathbb{X}}[\mathcal{E}(X_{\bar{r}}{)}_{{\mathbf{S}}_r}\mathcal{E}(X_{\bar{r}}{)}_{{\mathbf{S}}_r}^{⊺}{]}^{-1}{\mathbb{E}}_{\mathbb{X}}[\mathcal{E}(X_{\bar{r}}{)}_{{\mathbf{S}}_r}\mathcal{E}(X_r{)}^{⊺}]\end{array}$$

$$\Vert \text{e}{\Vert }_{\infty }\le 2\sigma$$

$$\Vert {\mathbb{E}}_{\mathbb{X}}[|\text{e}|]{\Vert }_{\infty }\le \mu .$$

$$\begin{array}{r}\hat{L}(\text{W})\triangleq \frac{1}{2N}\Vert \mathcal{E}({\text{X}}^r)-\mathcal{E}({\text{X}}^{\bar{r}})\text{W}{\Vert }_F^2\end{array}$$

$$\hat{f}(\text{W})$$

$$\hat{\text{W}}=\underset{\text{W}}{\min }\hat{f}(\text{W}).$$

$$\begin{array}{rl}\nabla {\hat{L}}_{\text{W}}(\text{W})& =\frac{1}{N}{\mathcal{E}({\text{X}}^{\bar{r}})}^{⊺}\mathcal{E}({\text{X}}^{\bar{r}})\text{W}-\frac{1}{N}{\mathcal{E}({\text{X}}^{\bar{r}})}^{⊺}\mathcal{E}({\text{X}}^r)\\ {\nabla }^2{\hat{L}}_{\text{W}}(\text{W})& =\left[\begin{array}{cccc}\hat{\text{H}}& 0& \dots & 0\\ 0& \hat{\text{H}}& \dots & 0\\ ⋮& ⋮& \dots & ⋮\\ 0& 0& \dots & \hat{\text{H}}\end{array}\right]\end{array}$$

$$\displaystyle\begin{split}\textbf{H}=\mathbb{E}_{\mathbb{X}}\big{[}\mathcal{E}(X_{\bar{r}})\mathcal{E}(X_{\bar{r}})^{\intercal}\big{]}\ .\end{split}$$

$${\mathbf{S}}_r=\bigcup _{i\in {\mathbf{\pi }}_r\cup {\mathbf{c}}_r}\{{\rho }_{[i-1]}+1,\dots ,{\rho }_{[i]}\}$$

$${\mathbf{S}}_r^c=[{\rho }_{[n]}]-{\mathbf{S}}_r-\{{\rho }_{[r-1]}+1,\dots ,{\rho }_{[r]}\}$$

$$\displaystyle\begin{split}&\hat{\lambda}>\frac{4}{\alpha}\sqrt{\frac{\rho_{r}}{N}}\max\big{(}(1-\alpha)(\sqrt{2\sigma^{2}\log(\rho_{\bm{\pi}_{r}\cup\mathbf{c}_{r}}\rho_{r})}+\mu)\\ &,\sqrt{\bar{\rho}}(\sqrt{2\sigma^{2}\log(|(\bm{\pi}_{r}\cup\mathbf{c}_{r})^{c}|\bar{\rho}\rho_{r})}+\mu)\big{)}\end{split}$$

$$=\frac{{\sum }_{r=1}^n|(\widehat{{\mathbf{\pi }}_r}\cup \widehat{{\mathbf{c}}_r})\cap ({\mathbf{\pi }}_r\cup {\mathbf{c}}_r)|}{{\sum }_{r=1}^n|(\widehat{{\mathbf{\pi }}_r}\cup \widehat{{\mathbf{c}}_r})|}$$

$$=\frac{{\sum }_{r=1}^n|(\widehat{{\mathbf{\pi }}_r}\cup \widehat{{\mathbf{c}}_r})\cap ({\mathbf{\pi }}_r\cup {\mathbf{c}}_r)|}{{\sum }_{r=1}^n|({\mathbf{\pi }}_r\cup {\mathbf{c}}_r)|}$$

$$F_1\text{-score}=\frac{2\times \text{Precision}\times \text{Recall}}{\text{Precision}+\text{Recall}}$$

$$L({\text{W}}_{{\mathbf{S}}_r.})\triangleq \frac{1}{2}\mathbb{E}[\sum _{i=1}^{m_r-1}(\mathcal{E}(X_{\bar{r}}{)}_{{\mathbf{S}}_r}^{⊺}{\text{W}}_{{\mathbf{S}}_{r}i}-\mathcal{E}(X_r{)}_i{)}^2]$$

$$g(t)$$

$$\frac{d^{2}g(t)}{d{t}^2}$$

​E(xrˉ1​)⊺E(xrˉ2​)⊺⋮E(xrˉ∏i∈πr​∪cr​​mi​​)⊺​</