Induction of Interpretable Possibilistic Logic Theories from Relational Data

TL;DR

This paper introduces a new method for learning interpretable relational models using possibilistic logic, which results in faster, more transparent theories compared to traditional Markov Logic Networks.

Contribution

The paper presents a novel SRL approach that encodes relational models as stratified classical theories with explicit certainty levels, enhancing interpretability and efficiency.

Findings

Faster learning compared to Markov Logic Networks

Produces more interpretable models

Explicit encoding of certainty levels in theories

Abstract

The field of Statistical Relational Learning (SRL) is concerned with learning probabilistic models from relational data. Learned SRL models are typically represented using some kind of weighted logical formulas, which make them considerably more interpretable than those obtained by e.g. neural networks. In practice, however, these models are often still difficult to interpret correctly, as they can contain many formulas that interact in non-trivial ways and weights do not always have an intuitive meaning. To address this, we propose a new SRL method which uses possibilistic logic to encode relational models. Learned models are then essentially stratified classical theories, which explicitly encode what can be derived with a given level of certainty. Compared to Markov Logic Networks (MLNs), our method is faster and produces considerably more interpretable models.

Tables

Table 1. Table 1: The possibilistic logic theory learned in the Yeast-Proteins dataset, not showing the hard rules and actual weights (but note that λ ⊥ < λ 1 < … < λ 5 subscript 𝜆 bottom subscript 𝜆 1 … subscript 𝜆 5 \lambda_{\bot}<\lambda_{1}<...<\lambda_{5} ). All rules are implicitly constrained by AllDiff constraints.

…  (112 hard constraints not shown here)
$\left(\text{Complex}(A,B)\leftarrow \text{ProteinClass}(A,C),\text{Interaction}(D,A),\text{Complex}(D,B),\wedge \text{ProteinClass}(D,C),{\lambda }_5\right)$
$\left(\text{Phenotype}(A,B)\leftarrow \text{Interaction}(C,A)\wedge \text{ProteinClass}(A,D)\wedge \text{Phenotype}(C,B)\wedge \text{ProteinClass}(C,D),{\lambda }_4\right)$
$\left(\text{ProteinClass}(A,B)\leftarrow \text{ProteinClass}(D,B)\wedge \text{Complex}(A,C)\wedge \text{Complex}(D,C),{\lambda }_3\right)$
$\left(\text{Enzyme}(A,B)\leftarrow \text{ProteinClass}(A,C)\wedge \text{Interaction}(A,D)\wedge \text{Enzyme}(D,B),\text{ProteinClass}(D,C),{\lambda }_2\right)$
$\left(\text{Location}(A,B)\leftarrow \text{Location}(D,B)\wedge \text{Complex}(A,C)\wedge \text{Complex}(D,C),{\lambda }_1\right)$
$(\perp ,{\lambda }_{\perp })$

Table 2. Table 2: The possibilistic logic theory learned in the UWCSE dataset (here λ i ≤ λ i + 1 subscript 𝜆 𝑖 subscript 𝜆 𝑖 1 \lambda_{i}\leq\lambda_{i+1} , hard rules of the theory are not shown and the variables in the rules are implicitly constrained by AllDiff constraints).

…  (116 hard constraints not shown here)
$\left(\text{Publication}(V0,V1)\leftarrow \text{TempAdvisedBy}(V3,V1)\wedge \text{Publication}(V0,V3),{\lambda }_{16}\right)$
$\left(\text{PostQuals}\left(V0\right)\leftarrow \text{TempAdvisedBy}(V0,V3)\wedge \text{TempAdvisedBy}(V4,V3)\wedge \text{PostQuals}\left(V4\right),{\lambda }_{15}\right)$
$\left(\text{AdvisedBy}(V0,V1)\leftarrow \text{FacultyAffiliate}\left(V1\right),\text{Publication}(V3,V1)\wedge \text{Publication}(V3,V0),{\lambda }_{15}\right)$
$\left(\text{PostGenerals}\left(V0\right)\leftarrow \text{Publication}(V2,V0)\wedge \text{AdvisedBy}(V0,V4),{\lambda }_{14}\right)$
$\left(\text{FacultyAdjunct}\left(V0\right)\leftarrow \text{AdvisedBy}(V2,V0),\text{FacultyAdjunct}\left(V4\right)\wedge \text{AdvisedBy}(V2,V4),{\lambda }_{13}\right)$
$\left(\text{PostGenerals}\left(V0\right)\leftarrow \text{TaughtBy}(V2,V0,V4)\wedge \text{Student}\left(V0\right),{\lambda }_{13}\right)$
$\left(\text{PostQuals}\left(V0\right)\leftarrow \text{AdvisedBy}(V0,V3)\wedge \text{ProjectMember}(V4,V3),{\lambda }_{12}\right)$
$\left(\text{AdvisedBy}(V0,V1)\leftarrow \text{Publication}(V3,V1)\wedge \text{FacultyAdjunct}\left(V1\right)\wedge \text{Publication}(V3,V0),{\lambda }_{11}\right)$
$\left(\text{PostQuals}\left(V0\right)\leftarrow \text{AdvisedBy}(V0,V3)\wedge \text{PostQuals}\left(V4\right)\wedge \text{AdvisedBy}(V4,V3),{\lambda }_{10}\right)$
$\left(\text{Publication}(V0,V1)\leftarrow \text{FacultyAffiliate}\left(V4\right),\text{FacultyAffiliate}\left(V1\right)\wedge \text{Publication}(V0,V4),{\lambda }_9\right)$
$\left(\text{PostGenerals}\left(V0\right)\leftarrow \text{AdvisedBy}(V0,V3)\wedge \text{PostGenerals}\left(V4\right)\wedge \text{AdvisedBy}(V4,V3),{\lambda }_8\right)$
$\left(\text{Faculty}\left(V0\right)\leftarrow \text{TempAdvisedBy}(V2,V0),{\lambda }_7\right)$
$\left(\text{PostQuals}\left(V0\right)\leftarrow \text{AdvisedBy}(V0,V3)\wedge \text{FacultyAdjunct}\left(V3\right),{\lambda }_6\right)$
$\left(\text{Faculty}\left(V0\right)\leftarrow \text{AdvisedBy}(V2,V0),{\lambda }_5\right)$
$\left(\text{Publication}(V0,V1)\leftarrow \text{FacultyAdjunct}\left(V4\right),\text{FacultyAdjunct}\left(V1\right)\wedge \text{Publication}(V0,V4),{\lambda }_4\right)$
$\left(\text{PreQuals}\left(V0\right)\leftarrow \text{TempAdvisedBy}(V0,V3),{\lambda }_3\right)$
$\left(\text{FacultyAdjunct}\left(V0\right)\leftarrow \text{Publication}(V2,V4),\text{PreQuals}\left(V4\right)\wedge \text{Publication}(V2,V0),{\lambda }_2\right)$
$\left(\text{PostQuals}\left(V0\right)\leftarrow \text{AdvisedBy}(V0,V3),{\lambda }_1\right)$
$(\perp ,{\lambda }_{\perp })$

Table 3. Table 3: The possibilistic logic theory learned in the UWCSE dataset (here λ i ≤ λ i + 1 subscript 𝜆 𝑖 subscript 𝜆 𝑖 1 \lambda_{i}\leq\lambda_{i+1} , hard rules of the theory are not shown and the variables in the rules are implicitly constrained by AllDiff constraints).

0	$\text{FacultyEmeritus}\left(V1\right)$
-5.64478	$\text{AdvisedBy}(V1,V2)$
-5.10868	$\text{Student}\left(V1\right)$
-2.06003	$\text{FacultyAffiliate}\left(V1\right)$
-3.18786	$\text{PostQuals}\left(V1\right)$
-3.64133	$\text{PostGenerals}\left(V1\right)$
-0.80596	$\text{FacultyAdjunct}\left(V1\right)$
-0.0391718	$\text{TempAdvisedBy}(V1,V1)$
-4.12274	$\text{Publication}(V1,V2)$
-0.0551981	$\text{AdvisedBy}(V1,V1)$
-2.95067	$\text{Faculty}\left(V1\right)$
-6.56008	$\text{ta}(V1,V2,V3)$
-4.90068	$\text{ProjectMember}(V1,V2)$
-4.44769	$\text{TempAdvisedBy}(V1,V2)$
5.10868	$\text{Professor}\left(V1\right)$
-6.31879	$\text{TaughtBy}(V1,V2,V3)$
-4.69704	$\text{PreQuals}\left(V1\right)$
1.67999e-06	$\neg \text{TaughtBy}(V1,V2,V3)\vee \neg \text{FacultyAdjunct}\left(V2\right)\vee \neg \text{TempAdvisedBy}(V2,V4)$
1.08275	$\neg \text{TaughtBy}(V1,V2,V3)\vee \neg \text{FacultyAdjunct}\left(V2\right)\vee \neg \text{Publication}(V4,V2)$
6.07986	$\text{Faculty}\left(V1\right)\vee \neg \text{TempAdvisedBy}(V2,V1)\vee \text{FacultyAffiliate}\left(V1\right)$
2.48337	$\text{PreQuals}\left(V1\right)\vee \text{PostQuals}\left(V1\right)\vee \text{ProjectMember}(V2,V1)\vee \neg \text{TempAdvisedBy}(V1,V3)$
9.33506	$\text{PreQuals}\left(V1\right)\vee \text{PostQuals}\left(V1\right)\vee \neg \text{AdvisedBy}(V1,V2)\vee \text{PostGenerals}\left(V1\right)$
-12.0071	$\neg \text{Student}\left(V1\right)\vee \text{FacultyAdjunct}\left(V1\right)\vee \text{Professor}\left(V1\right)\vee \text{Faculty}\left(V1\right)\vee \text{FacultyAffiliate}\left(V1\right)$
-2.05975	$\text{PostQuals}\left(V1\right)\vee \text{FacultyAdjunct}\left(V1\right)\vee \neg \text{Faculty}\left(V1\right)\vee \neg \text{AdvisedBy}(V2,V1)\vee \text{FacultyAffiliate}\left(V1\right)$

Equations

$$-5.11:$$

$$5.11:$$

$$-12.01:$$

$$\vee \text{faculty}(X)\vee \text{faculty-aff}(X)$$

$$p_{\mathcal{M}}(\omega )=\frac{1}{Z}\text{mbox}exp\left(\sum _{w:F\in \mathcal{M}}w{n}_F(\omega )\right)$$

$${\pi }_{\Theta }(\omega )=\min \{1-\lambda |(\alpha ,\lambda )\in \Omega ,\omega /\models \alpha \}$$

$${\rm Y}⟨\{\text{alice},\text{bob}\}⟩=$$

$$\{\text{alice},\text{bob}\})$$

$${\rm Y}[\{\text{alice},\text{bob}\}]=$$

$$(\{\text{fr}(2,1),\text{fr}(1,2),\text{sm}(2)\},\{1,2\})\}$$

$$P_{{\rm Y},k}(\alpha )=\sum _{\omega :\omega \models \alpha }{P}_{{\rm Y},k}(\omega )$$

$${\Theta }_{{\rm Y},k}=\left\{\left(\neg \alpha ,1-\frac{1}{c(\alpha )}{P}_{{\rm Y},k}(\alpha )\right)|\alpha \in g_k({\rm Y})\right\}$$

$${\lambda }_1^{\prime }\le {\lambda }_2^{\prime }\le \cdots \le {\lambda }_n^{\prime }$$

$$\sum _{i=1}^k(1-{\lambda }_i^{\prime })\cdot \left(|M_{i+1}|-|M_i|\right)=1$$

Full text

1 Introduction

The aim of Statistical Relational Learning (SRL) is to learn models that can make predictions from sets of relational facts. Many popular SRL frameworks, such as Markov Logic Networks (MLNs Richardson and Domingos (2006)), probabilistic soft logic Bach et al. (2015), and various forms of probabilistic logic programs Raedt and Kimmig (2015), use weighted logical formulas to encode the statistical regularities that have been observed in training data. Despite the use of logical formulas, learned models are often surprisingly hard to interpret. Consider, for instance, the following fragment of an MLN that was learned from the UWCSE dataset111https://alchemy.cs.washington.edu/data/uw-cse/:

[TABLE]

The first two formulas intuitively mean that, all things being equal, a given individual is unlikely to be a student and likely to be a professor. However, if a given individual is a professor, then the third formula becomes satisfied. Due to the large negative weight of this formula, it turns out that being a student is actually considered to be more likely than being a professor. In practice, there can be many formulas that interact in such a way, making it hard to predict the behavior of the MLN by inspecting the weighted formulas. This limits the usefulness of MLNs for explorative data analysis, and makes it almost impossible for domain experts to manually tweak a learned MLN. Probabilistic logic programming (PLP) languages, such as ProbLog, attach probabilities to either rules or facts. For programs with neither negation as failure nor cyclic dependencies, the individual probabilities have a clearer meaning than in MLNs in that a rule such as $0.5$ :: $a$ :- $b$ can be interpreted as if $b$ is true, $a$ is true with at least 50% probability. However, using negation as failure and cyclic dependencies often leads to counter-intuitive PLPs where rules cannot be understood in isolation (e.g. see Section 8.3 in Buchman and Poole (2017)). Yet, even for propositional PLPs, excluding negation as failure limits the expressivity of the language Buchman and Poole (2017). While the interpretability of AI systems is becoming increasingly important Baehrens et al. (2010); Sanchez et al. (2015); Ribeiro et al. (2016), we are not aware of any existing methods for learning joint relational models that focus on interpretability.

Possibilistic logic Lang et al. (1991) also uses weighted formulas, usually written as $(\alpha,\lambda)$ with $\alpha$ a classical formula and $\lambda\in[0,1]$ a certainty weight. As suggested in Kuželka et al. (2016), we can use possibilistic logic to encode probability distributions. The formula $(\alpha,\lambda)$ then expresses the constraint that the probability of any world violating $\alpha$ can be at most $1-\lambda$. Because of this constraint based semantics, formulas can safely be interpreted in isolation from the rest of the theory, which we believe is crucial for interpretability. The method proposed in Kuželka et al. (2016) derives a possibilistic logic theory from a density estimation tree Ram and Gray (2011), which is in turn learned from a set of training examples. Compared to Markov Random Fields (MRFs), the possibilistic logic theories resulted in a higher accuracy for Maximum A Posteriori (MAP) queries with small evidence sets, while MRFs were more accurate for larger evidence sets. Essentially, inference from possibilistic logic theories captures the conclusions that we can obtain by applying a form of commonsense reasoning (see Kuželka et al. (2015) for a theoretical justification of this view). In the presence of large amounts of evidence, however, MRFs can make predictions even when there is no obvious “default knowledge” that applies, by aggregating large amounts of individually weak and/or conflicting pieces of evidence.

In this paper we introduce a method for learning possibilistic logic theories from relational data. In principle, such theories could be learned by “lifting” the approach from Kuželka et al. (2016) to the relational setting. However, this technique relies on identifying a set of formulas $\alpha_{1},...,\alpha_{n}$ which are mutually exclusive and jointly exhaustive (corresponding to the branches of the density estimation tree). In a relational setting, this essentially requires us to enumerate isomorphism classes, of which there are typically exponentially many. As a result, the possibilistic logic theories we obtain quickly become prohibitively large (even though these theories could subsequently be pruned). Therefore, we follow a different strategy in this paper. To obtain suitable formulas, we first learn a set of hard constraints. These hard constraints allow us to generate non-trivial negative examples, which together with the positive examples obtained from the training data, allow us to learn a set of Horn rules that describe how the different predicates relate to each other. The restriction to Horn rules increases the interpretability of the learned theories, and leads to theories that are optimized for predicting positive literals, which is usually what is needed in applications. Note that the hard constraints are not restricted to Horn rules, which means that our theories can still be used to predict negative literals. In the last step, we use a form of relational model counting to associate a weight with each of the learned Horn rules.

To the best of our knowledge, our approach is the first that represents joint relational models in such a way that each weighted formula can be interpreted in isolation. The most closely related work is Serrurier and Prade (2007), where first-order possibilistic logic theories were learned in an Inductive Logic Programming (ILP) setting. However, the learned theories from that work are aimed at predicting a single target predicate. Moreover, their approach was based on a non-standard semantics for possibilistic logic, in which formulas cannot be interpreted in isolation. Finally, their approach is purely qualitative, i.e. formulas are ranked but are not given weights with a probabilistic interpretation.

2 Preliminaries

Throughout the paper, we consider a function-free first-order logic language $\mathcal{L}$, which is built from a set of constants Const, variables Var and predicates $\textit{Rel}=\bigcup_{i}\textit{Rel}_{i}$, where $\textit{Rel}_{i}$ contains the predicates of arity $i$. For $a_{1},...,a_{k}\in\textit{Const}\cup\textit{Var}$ and $R\in\textit{Rel}_{k}$, we call $R(a_{1},...,a_{k})$ an atom. If $a_{1},..,a_{k}\in\textit{Const}$, this atom is called ground. A literal is an atom or the negation of an atom, and a clause is a disjunction of literals. The formula $\alpha_{0}$ is called a grounding of $\alpha$ if $\alpha_{0}$ can be obtained from $\alpha$ by substituting each variable by a particular constant from Const. A formula is called closed if all variables are bound by a quantifier. A possible world $\omega$ is defined as a set of ground atoms. The satisfaction relation $\models$ is defined in the usual way.

A Markov logic network (MLN) Richardson and Domingos (2006) is a set of weighted formulas $w:F$, with $w\in\mathbb{R}$ and $F$ a function-free and quantifier-free first-order formula. The semantics are defined w.r.t. the groundings of the first-order formulas, relative to some finite set of constants. An MLN is seen as a template that defines an MRF. Specifically, an MLN $\mathcal{M}$ induces the following probability distribution on the set of possible worlds $\omega$:

[TABLE]

where $n_{F}(x)$ is the number of groundings of $F$ that are satisfied in $\omega$, and $Z$ is a normalization constant to ensure that $p_{\mathcal{M}}$ is a probability distribution. A key inference task for MLNs is computing the Maximum A Posteriori (MAP) consequences, i.e. determining which ground atoms are true in the most probable models of a given set of ground atoms. Formally, $(\mathcal{M},E)\models\alpha$, for $E$ a set of ground atoms and $\alpha$ a ground atom, iff $\forall\omega\,.\,(p_{\mathcal{M}}(\omega)=\max_{\omega^{\prime}}p_{\mathcal{M}}(\omega^{\prime}))\Rightarrow(\omega\models\alpha)$.

A possibilistic logic theory Lang et al. (1991) is a set of weighted formulas $(\alpha,\lambda)$ with $\alpha$ a propositional formula and $\lambda\in[0,1]$. A possibilistic logic theory $\Theta$ induces a mapping $\pi:\Omega\rightarrow[0,1]$, with $\Omega$ the set of propositional interpretations, which is defined for $\omega\in\Omega$ as:

[TABLE]

The distribution $\pi_{\Theta}$ is called a possibility distribution. The possibilistic logic theories we consider will be constructed such that $\sum_{\omega}\pi_{\Theta}(\omega)=1$, in which case $\pi_{\Theta}$ can be interpreted as a probability distribution. There is a common inconsistency-tolerant inference relation in possibilistic logic, which is actually the direct counterpart of MAP inference. Specifically, for $E$ a set of propositional formulas and $\alpha$ a propositional formula, we write $(\Theta,E)\models\alpha$ if $\forall\omega\,.\,(\pi_{\Theta}(\omega)=\max_{\omega^{\prime}}\pi_{\Theta}(\omega^{\prime}))\Rightarrow(\omega\models\alpha)$. Interestingly, the formulas $\alpha$ which are entailed in this sense can easily be determined syntactically. In particular, for $\mu\in[0,1]$ let $\Theta_{\mu}=\{\alpha\,|\,(\alpha,\lambda)\in\Theta,\lambda\geq\mu\}$. Let $\mu_{0}$ be the smallest threshold for which $\Theta_{\mu_{0}}\cup E$ is consistent. Then $(\Theta,E)\models\alpha$ iff $\Theta_{\mu_{0}}\cup E\models\alpha$. Hence inference in possibilistic logic can straightforwardly be implemented using a SAT solver.

In this paper we will learn possibilistic logic theories with first-order formulas instead of propositional formulas. Like MLNs, these first-order possibilistic logic theories should simply be seen as templates for normal (propositional) possibilistic logic theories that are obtained by replacing each weighted first-order formula $(\alpha,\lambda)$ by the formulas $(\alpha_{1},\lambda),...,(\alpha_{k},\lambda)$, with $\alpha_{1},...,\alpha_{k}$ the groundings of $\alpha$. It is easy to see that when $p_{\mathcal{M}}=\pi_{\Theta}$ for an MLN $\mathcal{M}$ and first-order possibilistic logic theory $\Theta$, it holds that $(\mathcal{M},E)\models\alpha$ iff $(\Theta,E)\models\alpha$. Kuželka et al. (2015) demonstrated how to construct a possibilistic logic theory $\Theta$ from a given MLN $\mathcal{M}$, such that $p_{\mathcal{M}}=\pi_{\Theta}$. However, the resulting possibilistic logic theory is exponential in size. In practice, the possibilistic logic theories we learn from data can thus only approximate what could be encoded in an MLN. This makes MLNs potentially better equipped to make predictions from large amounts of evidence, while making possibilistic logic less prone to making spurious predictions in situations where the amount of evidence is more limited.

3 Relational Marginals

In the context of SRL, we are typically given a large set of ground atoms $\mathcal{A}$ as training data. This set essentially corresponds to a single example of a relational structure. Intuitively, we want to learn a probability distribution over such relational structures, but we clearly cannot estimate such a distribution from one example. The solution we propose is to construct a large number of training examples by sampling small fragments of this global relational structure, and then estimating a probability distribution over these fragments. We will refer to $\Upsilon=(\mathcal{A},\mathcal{C})$, with $\mathcal{C}$ the set of constants appearing in $\mathcal{A}$, as an example. We now explain how we can obtain a collection of “local” training examples, which will correspond to (isomorphism classes of) fragments of this “global” example.

Definition 1**.**

A (global) example is a pair $(\mathcal{A},\mathcal{C})$, with $\mathcal{C}$ a set of constants and $\mathcal{A}$ a set of ground atoms which only use constants from $\mathcal{C}$. Let $\Upsilon=(\mathcal{A},\mathcal{C})$ be an example and $\mathcal{S}\subseteq\mathcal{C}$. The fragment $\Upsilon\langle S\rangle=(\mathcal{B},\mathcal{S})$ is defined as the restriction of $\Upsilon$ to the constants in $\mathcal{S}$, i.e. $\mathcal{B}$ is the set of all atoms from $\mathcal{A}$ which only contain constants from $\mathcal{S}$.

Intuitively, we can repeatedly sample subsets $\mathcal{S}$ and then consider each $\Upsilon\langle S\rangle$ as a training example. However, the constants appearing in each of these fragments will be different, hence to enable generalization we need to consider their isomorphism classes.

Definition 2** (Isomorphism).**

Two examples $\Upsilon_{1}=(\mathcal{A}_{1},\mathcal{C}_{1})$ and $\Upsilon_{2}=(\mathcal{A}_{2},\mathcal{C}_{2})$ are isomorphic, denoted as $\Upsilon_{1}{\approx}\Upsilon_{2}$, if there exists a bijection $\sigma:\mathcal{C}_{1}\rightarrow\mathcal{C}_{2}$ such that $\sigma(\mathcal{A}_{1})=\mathcal{A}_{2}$, where $\sigma$ is extended to ground atoms in the usual way.

Definition 3** (Local example).**

Let $k\in\mathbb{N}$ and let $\mathcal{L}_{k}$ be the language which contains the same predicates and variables as $\mathcal{L}$ but only constants from the set $\{1,2,...,k\}$. A local example of width $k$ is a pair $\omega=(\mathcal{A},\{1,...,k\})$, where $\mathcal{A}$ is a set of ground atoms from the language $\mathcal{L}_{k}$. For an example $\Upsilon=(\mathcal{A},\mathcal{C})$ and $\mathcal{S}\subseteq\mathcal{C}$, we write $\Upsilon[\mathcal{S}]$ for the set of all local examples of width $|\mathcal{S}|$ which are isomorphic to $\Upsilon\langle\mathcal{S}\rangle$.

To distinguish local examples from global examples, we will denote them using lower case Greek letters such as $\omega$ instead of upper case letters such as $\Upsilon$.

Example 1**.**

For $\Upsilon=(\{\textit{fr}(\textit{alice},\textit{bob}),\allowbreak\textit{fr}(\textit{bob},\textit{alice}),\allowbreak\textit{fr}(\textit{bob},\allowbreak\textit{eve}),\allowbreak\textit{fr}(\textit{eve},\textit{bob}),\allowbreak\textit{sm}(\textit{alice})\},\allowbreak\{\textit{alice},\allowbreak\textit{bob},\allowbreak\textit{eve}\})$ we have:

[TABLE]

We can now naturally define a probability distribution over local examples of width $k$.

Definition 4** (Relational marginal distribution).**

Let $\Upsilon=(\mathcal{A},\mathcal{C})$ be an example and $k\in\mathbb{N}$. The relational marginal distribution of $\Upsilon$ of width $k$ is a distribution $P_{\Upsilon,k}$ over local examples, where $P_{\Upsilon,k}(\omega)$ is defined as the probability that $\omega$ is sampled by the following process:

Uniformly sample a subset $\mathcal{S}$ of $k$ constants from $\mathcal{C}$. 2. 2.

Uniformly sample a local example $\omega$ from the set $\Upsilon[\mathcal{S}]$.

For a closed formula $\alpha$, we also define:

[TABLE]

In the following, constant-free existentially-quantified conjunctions of atoms will play an important role, as they are the syntactic counterpart of the isomorphism classes $\Upsilon[\mathcal{S}]$. For such a conjunction $\alpha$, it holds that $P_{\Upsilon,k}(\alpha)$ is equal to the probability that a randomly sampled set $\mathcal{S}$ of $k$ constants satisfies $\Upsilon\langle\mathcal{S}\rangle\models\alpha$. In this sense, relational marginal distributions faithfully model the probabilities of isomorphism classes of local examples. Naturally, other probability distributions on local examples might also faithfully model the probabilities of these isomorphism classes, but it is easy to see that relational marginal distributions have the highest entropy among such models.

The idea of relational marginals is similar to the random selection semantics used in Schulte et al. (2014), but the difference is that for relational marginals, we restrict the sample sets to have fixed cardinality and then standardize them as local examples. This allows us to construct a standard probability distribution over local examples.

4 Possibilistic Logic Encoding of Relational Marginals

In this section we describe how relational marginals can be encoded in possibilistic logic. As we show first, in principle we can use a direct generalization of the approach from Kuželka et al. (2016), by taking advantage of the fact that each isomorphism class $\Upsilon[\mathcal{S}]$ of local examples corresponds to a constant-free existentially-quantified conjunction of atoms. For an example $\Upsilon$, let $g_{k}(\Upsilon)=\{\alpha_{1},...,\alpha_{n}\}$ be a set that contains one such formula for each isomorphism class of local examples of width $k$.

Definition 5** (Possibilistic encoding of relational marginals).**

Let $\Upsilon$ be an example and let $k\in\mathbb{N}$. The possibilistic logic theory corresponding to $P_{\Upsilon,k}$ is defined as

[TABLE]

where $c(\alpha)$ is the cardinality of the isomorphism class represented by $\alpha$.

Proposition 1**.**

Let $\Upsilon$ be an example, $k\in\mathbb{N}$, and $\omega$ a local example of width $k$. It holds that $P_{\Upsilon,k}(\omega)=\pi(\omega)$ where $\pi(.)$ is the possibility distribution associated with $\Theta_{\Upsilon,k}$.

Proof.

Let $\omega$ be a local example of width $k$. By definition, $g_{k}(\Upsilon)$ contains a unique formula $\alpha^{*}$ such that $\omega\models\alpha^{*}$, since the formulas in $g_{k}(\Upsilon)$ define a partition of local examples into isomorphism classes. Accordingly, $\neg\alpha^{*}$ is the unique formula appearing in $\Theta_{\Upsilon,k}$ which is not satisfied by $\omega$. By (2), we therefore have $\pi(\omega)=1-(1-P_{\Upsilon,k}(\alpha^{*})/c(\alpha^{*})=P_{\Upsilon,k}(\alpha^{*})/c(\alpha^{*})=P_{\Upsilon,k}(\omega)$, where the last equality hold because all local examples from the same partition class have the same probability in a relational marginal distribution. ∎

The number of isomorphism classes typically grows very quickly with increasing $k$, so the exact transformation from Definition 5 can only be used for very simple problem domains. In practice, representing the relational marginal distribution exactly is typically not feasible. An exact representation would moreover not necessarily generalize well to previously unseen data. Therefore, for the remainder of this paper, we will focus on learning approximate possibilistic logic representations of relational marginal distributions.

Specifically, our aim is to construct a possibilistic logic theory $\Theta=\{(\alpha_{1},\lambda_{1}),...,(\alpha_{n},\lambda_{n})\}$ such that for the associated possibility distribution $\pi$ it holds that $\pi(\omega)$ is approximately equal to $P_{\Upsilon,k}(\omega)$. This problem can be decomposed in two steps. The first step is structure learning, i.e. choosing suitable formulas $\alpha_{1},...,\alpha_{n}$. In this paper, we will only consider constant-free and quantifier-free formulas. However, recall that first-order possibilistic logic theories are seen as templates for propositional theories, which means that all variables in the formulas $\alpha_{1},...,\alpha_{n}$ are implicitly universally quantified. The second step is weight learning. In this step, we aim to find the weights $\lambda_{1},...,\lambda_{n}$ for which $\pi$, seen as a probability distribution, maximizes the likelihood of a set of training examples. Note that if $\lambda_{1}\leq...\leq\lambda_{n}$ we can assume w.l.o.g. that $\alpha_{1}=\bot$. We need to include such a formula $\alpha_{1}$ to encode the probability of the most probable worlds (which is then given by $1-\lambda_{1}$).

As the transformation from Definition 5 illustrates, weight learning becomes very simple when using mutually exclusive formulas. However, using mutually exclusive formulas is not desirable, as such formulas quickly become very large222One exception is when $k=1$, which corresponds to the propositional case, where density estimation trees can be used, as was proposed in Kuželka et al. (2016)., which also makes the resulting theories difficult to interpret. Therefore, in practice, we will rely on greedy methods for weight learning. These will be discussed in Section 6.

5 Structure Learning

In this section, we propose a method to learn Horn rules that can be used to predict all predicates from $\Upsilon$. Using Horn rules makes the resulting possibilistic logic theories more interpretable, and allows us to optimize them for predicting atoms, which is what is usually required. Learning Horn rules using methods based on inductive logic programming Muggleton and De Raedt (1994) typically requires both positive and negative training examples. In Subsection 5.1 we explain how to construct examples and then discuss our method for learning Horn rules in Subsection 5.2.

5.1 Constructing Training Examples

Constructing positive examples for a given predicate $P$ is straightforward: we can simply take all, or a subsample, of the true $P$-atoms from $\Upsilon$, Typically, there are significantly more negative examples than positive ones; e.g. in a typical social network there are many more examples of non-friends than of friends. Simply subsampling the negative examples is unlikely to be effective, as most of the resulting negative examples might be uninteresting, in the sense that they can be explained by some simple hard rules that hold for the domain. Hence, we first learn a set of such hard rules, and then only consider negative examples that are consistent with them.

We are interested in hard rules that are universally quantified, constant-free clauses with no counterexamples in $\Upsilon$. We find such clauses by exhaustively constructing all clauses (modulo isomorphism) containing at most $t$ literals and at most $k$ variables, where $k$ is the width of the relational marginal distribution and $t$ is a parameter of the method. For each clause, we check whether $\Upsilon\not\models\neg\alpha$ holds with a CSP solver. We store each such clause in a list if the list does not contain another clause that subsumes it. Because learning hard rules that only contain unary literals is typically easier than learning more general rules, we use a higher size limit $t^{\prime}>t$ for these rules.

Let $\Delta$ be the set of discovered hard rules, and $\Upsilon=(\mathcal{A},\mathcal{C})$ be the global example. To select negative training examples, we reject all samples $a$ for which $\bigwedge\mathcal{A}\wedge a\wedge\Delta$ does not have a model when grounded333We use incremental grounding for efficiency. over the set of constants $\mathcal{C}$. The result is a subsample of non-trivial negative examples. In addition, this process allows us to estimate the total number of non-trivial negative examples, which we use to compute the weight of the negative examples when estimating the accuracies of the Horn rules.

5.2 Learning Horn Rules

To find Horn rules, we employ a beam search method, which relies on two parameters: the size of the beam $b$ and the maximum number of literals in the body of a rule $l$. As before, $k$ is the width of the local examples. For a given target predicate $P$ of arity $m$, we initialize the list of candidate rules with the rule $P(X_{1},...,X_{m})\leftarrow\top$. In each iteration of the search, we construct all possible single-literal extensions of each rule in the beam such that the constraints on the number literals and variables are not violated. From these candidate rules, we select a set of non-isomorphic rules and evaluate their accuracy on the (weighted) sets of positive and negative examples. The algorithm then selects the $b$ most accurate rules to serve as the candidate rules for the next iteration. The algorithm terminates when no new candidate rules can be generated without violating the constraints on the number of literals and variables and returns the best found rule. This beam search method is repeated several times for each predicate $P$. Most rules found during one run of the beam search typically entail similar sets of examples. To promote diversity within each run of beam search, we discard rules that are subsumed by previously found rules.

We employ several well-known techniques to speed up the search. First, instead of checking isomorphism for every pair of candidate rules, we efficiently select non-isomorphic rules by hashing each one using a straightforward generalization of the Weisfeiler-Lehman labeling procedure Weisfeiler and Lehman (1968). Then, we only check if two rules are isomorphic if they have the same hash value, and if so one of them is removed. Second, the algorithm maintains a set Forbidden of minimal rules which entailed zero positive examples in the previous iterations of the beam search. Before evaluating new candidate rules, the algorithm discards candidate rules which are subsumed by a rule from the set Forbidden. Third, to reduce the negative plateau effect, known from relational learning Alphonse and Osmani (2008), we add to every constructed rule a literal $\textit{AllDiff}(V_{1},\dots,V_{k})$, which is true iff all variables in its argument are mapped to different terms. This also improves the interpretability of the rules.

6 Weight Learning

Let us first assume that an ordering of the formulas $(\alpha_{1}=\bot,\alpha_{2}\dots,\alpha_{n})$ is given, and we want to learn weights $\lambda_{1}\leq...\leq\lambda_{n}$ which maximize the likelihood of a set of local examples $\mathcal{E}$ that have been sampled from $P_{\Upsilon,k}$. These weights can be found by solving the following optimization problem:

• •

Variables: $\lambda_{1}^{\prime},\lambda_{2}^{\prime},\dots,\lambda_{n}^{\prime}$.

• •

Maximize: $\prod_{\omega\in E}P(\omega)=\prod_{i=1}^{n}(1-\lambda_{i}^{\prime})^{|\mathcal{E}_{i+1}|-|\mathcal{E}_{i}|}$ where $\mathcal{E}_{i}=\{\omega\in\mathcal{E}|\omega\models\alpha_{i}\wedge\dots\wedge\alpha_{n}\}$.

• •

Subject to:

[TABLE]

where $M_{i}=\{\omega|\omega\models\alpha_{i}\wedge\dots\wedge\alpha_{n}\}$ and (4) forces probabilities of all possible worlds to sum to $1$.

This optimization problem can be converted to a geometric programming problem, similar to the geometric programming encoding proposed in Kuželka et al. (2016). Note that geometric programming problems can be converted to convex programming problems by a change of variables, and can thus be solved using standard convex programming methods Boyd et al. (2007).

We can think of $\mathcal{E}$ as an IID sample from the set of local examples in the multi-set $\{\omega|\omega\in\Upsilon[\mathcal{S}],\mathcal{S}\subseteq\mathcal{C},|\mathcal{S}|=k\}$, where $\Upsilon=(\mathcal{A},\mathcal{C})$ is the given global example. However, assuming all $\alpha_{i}$ are constant-free, it is easy to check that we will get the same values (in expectation) of the parameters $|\mathcal{E}_{i}|$ if we instead use the set $\{\omega|\omega=\Upsilon\langle\mathcal{S}\rangle,\mathcal{S}\subseteq\mathcal{C},|\mathcal{S}|=k\}$.

Computing the parameters $|\mathcal{E}_{i}|$ and $|M_{i}|$ needed for weight learning is difficult (#P-hard), so the algorithm uses a greedy approach to search for the best ordering of the formulas. It starts with a possibilistic logic theory containing only the learned hard rules. It iteratively tries to add, at each possible positions, one rule $\alpha$ from the set of candidate rules found by the structure learning algorithm. If adding the rule $\alpha$ increases the likelihood score, we keep $\alpha$ in the theory, at the position that yielded the best improvement. This approach permits caching and reusing many of the parameter computations (i.e. the computed parameters for many cuts of the stratified theory will be the same for many iterations of the algorithm). During the learning process, the algorithm simplifies the constructed theories (using a relational SAT solver). It removes rules which are implied by other rules in the theory that have higher weights, and it also removes redundant literals from the individual rules.

7 Experiments

We compare our approach’s learned models to learned MLNs for various MAP inference tasks. We learned MLNs using the default structure learner in the Alchemy package Kok and Domingos (2005).444http://alchemy.cs.washington.edu/ For the MLNs, we used RockIt Noessner et al. (2013) to perform MAP inference.

7.1 Methodology

Our learning algorithm is implemented in Java and uses the SAT4j library Berre and Parrain (2010). Cryptominisat Soos (2010) is used for our implementation of relational version of the model counter Chakraborty et al. (2016). It uses the JOptimizer package to solve the geometric programming problems555http://www.joptimizer.com needed for the maximum likelihood estimation.

We use two standard SRL datasets: UWCSE and Yeast-Proteins.The UWCSE dataset described relations among students, professors, papers, subjects, terms and projects in the CS department of the University of Washington. This dataset contains among other the following relations (predicates) AdvisedBy/2, TempAdvisedBy/2, Publication/2, TaughtBy/3, TA/3, Student/1, Professor/1, PostQuals/1. This dataset is split into five groups: AI, language, theory, graphics, and systems. We use AI, language and theory as a training set and graphics and systems as a test set. The Yeast-Proteins dataset contains proteins and the relations among them. We use a version in which the interaction relation is symmetric. This dataset contains the following relations: Interacts/2, Enzyme/2, Complex/2, ProteinClass/2, Function/2, Phenotype and Location/2. We randomly divide the constants (entities) in this dataset into two disjoint sets of equal size. The training set consists of atoms containing only the constants from the first set and the test set contains only the constants from the second set. This ensures that no information leaks from the training set into the test set.

We evaluate the performance of the learned models as follows. For each $k=1,\dots,\textit{k}_{max}$, we sample a set of evidence literals from the test set. We then predict the MAP state by each of the learned models and compute the Hamming error, which measures the size of the symmetric difference of the predicted MAP world and the set of the literals in the test set. We then report the cumulative differences between the errors of the models, as this clearly highlights the overall trends.

7.2 Results

The possibilistic logic theory learned for the Yeast-Proteins dataset is shown in Table 1. The rules seem to encode meaningful relations that hold in the dataset. For instance, if a protein $A$ is contained in a complex $C$, another protein $D$ is in $C$ as well and $D$ is at location $B$ then $A$ is also at that location. The learned MLN, on the other hand, only contained rules that model the prior probabilities of the individual predicates and one additional rule that expresses the symmetry of the Interaction relation. Hence, the only type of prediction made by this MLN consists in computing the symmetric closure of the interaction literals, which is why we do not show any separate baseline prediction for this dataset. The possibilistic logic theory has lower Hamming errors for evidence sets up to around 1500 literals (see Figure 1, left panel), which can be seen from the fact that the cumulative difference is increasing over this range. For larger evidence sets, the possibilistic theory intuitively predicts “too much”, resulting in a higher Hamming error than the MLN predictions.

The theory which was learned for the UWCSE dataset is larger, and is therefore shown in the appendix, where we also show the corresponding learned MLN. The possibilistic logic theory again contains rules which are intuitive, capturing meaningful relations for this domain. The formulas in the MLN are much harder to interpret. As shown in Figure 1, the possibilistic logic theory again reaches smaller Hamming errors than the learned MLN for small evidence sets, in this case for evidence sets of up to about 250 literals (right panel). It is always better than the baseline which predicts everything not in the evidence as false (middle panel).

Inference in the possibilistic logic theories is, on average, substantially faster than MAP inference in the MLNs (using RockIt). For UWCSE, the speed-up was between one and two orders of magnitude. The possibilistic logic prediction only requires us to solve a logarithmic number of SAT queries, whereas computing MLN MAP predictions requires solving a weighted MAX-SAT problem.

8 Conclusions

We have proposed a method for learning relational possibilistic logic theories. These theories are seen as templates for constructing “ground” (i.e. standard propositional) possibilistic logic theories, similar to how Markov logic networks can be seen as templates for constructing Markov random fields. In particular, as in standard possibilistic logic, each weighted formula has an intuitive interpretation as a constraint on the probability distribution that is being modelled. To formally describe what this probability distribution represents, we have introduced the notion of a relational marginal distribution, which we can intuitively think of as a probability distribution over fixed-sized fragments of a given relational structure.

Our method jointly models the different predicates from the considered domain. This contrasts with inductive logic programming, which attempts to predict a single target predicate. Our approach to structure learning essentially boils down to applying ILP rule learning methods to each of the predicates in the domain. Another difference with the standard ILP setting is that we are not explicitly given a set of training examples in our setting. To generate such training examples, we explicitly learn a number of hard rules, and only consider negative examples that are non-trivial, in the sense that they cannot be explained by the hard rules alone. Interactions between the different rules are taken into account during a subsequent weight learning step.

The main design consideration of our method was to learn interpretable theories. However, as our experimental results have revealed, our method also leads to more accurate MAP predictions than Markov Logic Networks (MLNs) for small to moderately sized evidence sets. For larger evidence sets, MLNs lead to more accurate predictions, which is intuitively due to the fact that they are better equipped to aggregate large amounts of individually weak pieces of evidence. Inference in possibilistic logic is also considerably faster than methods for computing MAP queries from MLNs.

Acknowledgments

This work was supported by a grant from the Leverhulme Trust (RPG-2014-164) and ERC Starting Grant 637277. JD is partially supported by the KU Leuven Research Fund (C22/15/015), and FWO-Vlaanderen (G.0356.12, SBO-150033).

Appendix A Models Learned on UWCSE Dataset

Table 2 displays the possibilistic logic theory learned on the UWCSE dataset. The learned Markov logic network is displayed666We show the variables in the MLN as upper-case letters for consistency with the rest of the paper. However, note that it is more common for MLNs to use lower-case letters for variables. in Table 3.

Appendix B Algorithms for Parameter Estimation

In this section we provide additional details about algorithms for estimating the parameters of the maximum likelihood geometric programming problem stated in Section 6.

B.1 Subset Counting

Let $\alpha$ be an existentially quantified conjunction (i.e. a conjunctive query) and let us write $\textit{vars}(\alpha)$ for the set of variables occurring in $\alpha$. Then the $k$-extension of $\alpha$ is the conjunction $\exists V_{1}^{\prime},V_{2}^{\prime},\dots,V_{k}^{\prime}:\alpha\wedge\textit{card}(k,\allowbreak V_{1},\allowbreak\dots,\allowbreak V_{m},\allowbreak V_{1}^{\prime},\allowbreak\dots,\allowbreak V_{k}^{\prime})$ where $V_{1},\dots,V_{m}$ are the variables contained in $\textit{vars}(\alpha)$, $V_{1}^{\prime},\dots,V_{k}^{\prime}$ are new variables and the constraint $\textit{card}(k,\allowbreak V_{1},\allowbreak\dots,\allowbreak V_{m},\allowbreak V_{1}^{\prime},\allowbreak\dots,\allowbreak V_{k}^{\prime})$ is true iff the set $\{V_{1},\allowbreak\dots,\allowbreak V_{m},V_{1}^{\prime},\allowbreak\dots,\allowbreak V_{k}^{\prime}\}$ has cardinality $k$.

Definition 6** (Matching subset counting).**

Consider an example $\Upsilon=(\mathcal{A},\mathcal{C})$, $k\in\mathbb{N}$, and an existentially quantified conjunction $\alpha$. The matching subset counting problem is to count the cardinality of the set $\{\mathcal{S}|\mathcal{S}\subseteq\mathcal{C},|\mathcal{S}|=k,\Upsilon\langle\mathcal{S}\rangle\models\alpha\}$. The elements $\mathcal{S}$ of this set are called matching subsets for $\alpha$.

A naive solution would be to use a CSP solver Rossi et al. (2006) to obtain all tuples that make the $k$-extension of $\alpha$ satisfied, and to convert these into sets. In general, this approach actually has an optimal worst-case complexity when the CSP solver cannot do better than exhaustive search. However, its disadvantage is that even if the $k$-extension of $\alpha$ contained just $k$ variables, this method might discover each matching subset $k!$ times. Moreover, for conjunctive queries whose corresponding CSP problem can be solved faster than by brute-force search (e.g. for bounded hypertree-width formulas Gottlob et al. (2014)) this approach may not have worst-case optimality. Therefore, we now describe another approach which also relies on CSP solving, but which needs to generate fewer solutions per matching set in the worst case.

The pseudocode of our proposed algorithm for enumerating matching subsets is shown in Algorithm 1. The algorithm assumes access to a CSP solver which is capable of answering the following type of queries $\mathbf{CSP}(\alpha,V,\Upsilon)$: given a global example $\Upsilon=(\mathcal{A},\mathcal{C})$, a conjunctive query $\alpha$ and a variable $V\in\textit{vars}(\alpha)$, return the set of all constants $c\in\mathcal{C}$ such that $\mathcal{A}\models\alpha\theta$ where $\theta=\{V/c\}$ is a substitution. Such queries can either be implemented directly in the solver or by making multiple calls to an external CSP solver; in our implementation we have adopted the former approach. The algorithm uses these queries to iteratively build the set of matching subsets. It can be shown that it generates at most $2^{k}$ different solutions of the CSP problems per matching subset, which even for small values of $k$ is considerably better than the $k!$ solutions generated per matching subset by the naive method (e.g. $2^{6}=64$ whereas $6!=720$). For CSP problems from a tractable subclass, Algorithm 1’s runtime can be shown to be bounded by $O(2^{k}\cdot|T|\cdot f_{CSP}(k,n)+f_{pre}(k,n))$ where $T$ is the output, $n$ is the length of the input, $f_{CSP}(k,n)$ is the time needed for checking the existence of a solution of the CSP problem (after preprocessing) and $f_{pre}(k,n)$ is the time for preprocessing (e.g. for establishing global consistency).

For larger datasets, counting the number of matching subsets exactly is not feasible. Therefore we next describe an approximate counter, which relies on a recent algorithm called ApproxMC2 Chakraborty et al. (2016). ApproxMC2 is a method for propositional model counting. Since it uses SAT solvers as black-box oracles, it can be used for other problems as long as their solutions can be represented as fixed-length boolean vectors. In particular, it can be naturally used for approximate subset counting, since we can represent the $k$-subsets as Boolean vectors in which exactly $k$ entries are true. These entries correspond to indicator variables which are true if the corresponding constants are contained in the subset. To apply ApproxMC2, which relies on partitioning the space by long XOR constraints, all we need is to support the propagation of XOR constraints in the CSP solver. We propagate global XOR constraints using Gaussian elimination over GF[2]. We do not represent the indicator variables explicitly as decision variables of the CSP problem, but only in the equations over GF[2] inside the global constraint. If a constant is not present in domain of any variable then the corresponding indicator variable must be set to false. Likewise if a variable is assigned a constant $c$ then the corresponding indicator variable must be set to true. If an indicator variable is forced to be zero by Gaussian elimination, we can remove it from all domains.

In many cases, when the density of matching subsets is high, we can get reliable estimates in a much simpler way. In particular, we also implemented a third method which relies on uniformly drawing a small number of $k$-subsets from $\Upsilon$ and estimating the confidence intervals. If the relative size of the confidence interval is below a pre-fixed threshold, we use this value instead of employing the other, more costly methods.

Our overall approach is then to first try the exact method with a small maximum limit. If this fails, we use the simple sampling-based algorithm. If that fails as well, we try the exact method with larger maximum limit. Finally, if this fails as well, we use the algorithm based on ApproxMC2.

B.2 Model Counting

To compute the parameters of the optimization problem for likelihood maximization, we also need to compute model counts for the theories consisting of non-ground clauses over sets of $k$ constants. To this end, we implemented two algorithms based on the ApproxMC2 algorithm. The first version, which works best for small $k$, simply grounds the theory and computes the model count using ApproxMC2. The second version exploits the fact that the ApproxMC2 algorithm runs a SAT solver to decide if the theory has at most $t$ solutions (where $t$ is a threshold based on given tolerance parameters). It is not always necessary to ground the whole theory to check this. Therefore this second version uses a strategy based on cutting plane inference Riedel (2008), grounding the rules iteratively only when they become violated, where checking whether a non-ground rule is violated can be done using a CSP solver.