Properties and Extensions of Alternating Path Relevance - I

TL;DR

This paper explores the concept of alternating path relevance in logical theorem proving, presenting efficient graph-based computation methods and extensions to improve relevance filtering in large logical systems.

Contribution

It introduces a novel alternating path relevance concept, along with efficient algorithms and extensions for DPLL, enhancing relevance-based theorem proving.

Findings

Efficient graph-based algorithms for computing alternating path relevance.

Extensions to DPLL improve relevance filtering performance.

Results indicate effectiveness in large problem instances.

Abstract

When proving theorems from large sets of logical assertions, it can be helpful to restrict the search for a proof to those assertions that are relevant, that is, closely related to the theorem in some sense. For example, in the Watson system, a large knowledge base must rapidly be searched for relevant facts. It is possible to define formal concepts of relevance for propositional and first-order logic. Various concepts of relevance have been defined for this, and some have yielded good results on large problems. We consider here in particular a concept based on alternating paths.We present efficient graph-based methods for computing alternating path relevance and give some results indicating its effectiveness. We also propose an alternating path based extension of this relevance method to DPLL with an improved time bound, and give other extensions to alternating path relevance intended to improve its performance.

Full text

1 Introduction

In some applications, there may be knowledge bases containing thousands or even millions of assertions. Selecting relevant assertions has the potential to significantly reduce the cost of finding a proof of a desired conclusion from such large knowledge bases and even from smaller ones. Relevance can be defined in many ways. In first-order clausal theorem proving, a concept of relevance based on ”alternating” paths between clauses [PY03] permits relevant facts to be chosen automatically. Here we present efficient graph-based methods for computing alternating path relevance. We give some theoretical and practical results indicating its effectiveness. We also present a relationship between alternating path relevance and the set of support strategy [WRC65] in theorem proving. We incorporate this approach to relevance into the DPLL method. Finally, we present some extensions to alternating path relevance with a view to improving its effectiveness.

1.1 Related Work

Meng and Paulson [MP09] describe a relevance approach in which clauses are relevant if they share enough symbols with clauses that have previously been found to be relevant. They give clauses a “pass mark,” a number between 0 and 1. This is used as a filter, and the tests becomes increasingly strict as the distance from the conjecture increases. Their method makes use of two parameters, $p$ and $c$. They finally chose $p=0.6$ and $c=2.4$ based on experiments. This approach significantly improves the performance of the Sledgehammer theorem prover [MQP06] used with Isabelle [WPN08]. They also tried the alternating path relevance approach [PY03], but apparently did not set a bound on relevance distance, but rather included all of the relevant clauses.

Pudlak [Pud07] defines relevance in terms of finite models. The idea is to find a set ${\bf B}$ of clauses such that all of the finite models that have been constructed so far and satisfy ${\bf B}$, also satisfy the theorem $C$. Such a set ${\bf B}$ of clauses is a candidate for a sufficient set for proving the theorem. Clauses $F$ are added to ${\bf B}$ by constructing models of the theorem that do not satisfy ${\bf B}$, and finding clauses $F$ that contradict such models. This approach is attractive because humans seem to use semantics when proving theorem. It had good results on the bushy division of MPTP [Urb04], and was extended in the SRASS system [SP07] where it was among the most successful systems in the MPTP division. The SRASS system uses a syntactic similarity measure in addition to semantics, as an aid in ordering the axioms. However, the basic system [Pud07] appears to be inefficient in the presence of a large number of clauses. SRASS is also resource intensive for large theories.

The MaLARea system [Urb07] uses machine learning from previous proof attempts to guide the proof of new problems. It uses a Bayesian learning system to choose a relevance ordering of the axioms based on the symbols that appear in the conjecture. In each proof attempt, the $k$ most relevant clauses are used for various $k$, and various time limits are tried. It has given good performance on the ”chainy” division of MPTP, on which it solved more problems than E, SPASS, and SRASS. This system has been modified [USPV08] to take into account semantic features of the axioms in choosing the relevance ordering, similar to SRASS, and the combined system has outperformed both MaLARea and SRASS on the MPTP problems. The basic MaLARea system has been combined with neural network learning and has performed very well on the large theory batch division in the CASC competitions [Sut16] in 2017 and 2018. The Divvy system [RPS09] also orders the axioms, but does so solely on the basis of a syntactic similarity measure. For each proof attempt, a subset of the most relevant axioms is used. This system has obtained good results on the MPTP challenge problems.

The Sine Qua Non system [HV11] evaluates how closely two clauses are related by how many symbols they share, and which symbols they share. It also considers the length of paths between clauses as a measure of how closely they are related. The d-relevance idea of Sine Qua Non is straightforward, but in order to reduce the number of relevant axioms, a triggering technique is used. However, using the least common symbol as a trigger doesn’t work well, so they add a tolerance, a depth limit, and a generality threshold. Experiments revealed that some of these parameters don’t make much of a difference, and at least the depth limit turned out to be important. Tolerance was less important for the problems tried. This relevance measure can be computed very fast even for large sets of clauses. This approach has performed very well in the large theory division of CASC, and has been used by Vampire [RV99] and by some competing systems as well. However, sometimes clauses may be closely related in the Sine Qua Non system but not in the alternating path approach; for example, the literals $P(a,b)$ and $\neg P(b,a)$ are closely related in the former system but unrelated in the latter.

These systems do at least show that relevance can be effective in aiding theorem provers on large knowledge bases. Also, especially relevance measures that can be computed quickly may lead to spectacular increases in the effectiveness of theorem provers on very large knowledge bases. Even if the original knowledge base is small, large numbers of clauses may be generated during a proof attempt, and relevance techniques may help to select derived clauses leading to a proof.

2 Terminology

2.1 Connectedness of two clauses

This section contains basic definitions related to alternating paths. Some of the formalism is new. Standard definitions of terms, clauses, sets of clauses, substitutions, resolution, resolution proof, and satisfiability in first-order logic are assumed. If $A$ is an atom, then $A$ and $\neg A$ are literals, and each is the complement of the other. If $L$ is $\neg A$ then $\neg L$ will generally denote $A$.

The number of clauses in a set $S$ of first-order clauses is denoted by $|S|$.

The relation $\equiv$ on literals denotes syntactic identity. That is, it is the smallest relation on literals such that for all atoms $A$, $A\equiv A$, $\neg A\equiv\neg A$, $A\equiv\neg\neg A$, and $\neg\neg A\equiv A$. This treatment of negation permits one to say that $A$ and $B$ are complementary literals iff $A\equiv\neg B$.

A pair $L$ and $M$ of literals are complementary unifiable if there are substitutions $\alpha$ and $\beta$ such that $L\alpha\equiv\neg M\beta$.

An alternating path from $C_{1}$ to $C_{n}$ in a set $S$ of clauses is a sequence $C_{1},p_{1},C_{2},p_{2}$, $\dots,C_{n-1},p_{n-1},C_{n}$ where $C_{i}\in S$ for all $i$, $p_{i}$ is a pair $(L_{i},M_{i+1})$ of literals, $L_{i}\in C_{i}$, $M_{i+1}\in C_{i+1}$, $L_{i}$ and $M_{i+1}$ are complementary unifiable, and $L_{i}\not\equiv M_{i}$ for all $i$. Frequently the $p_{i}$ are omitted. Such a path is called alternating because it alternates between connecting literals in possibly different clauses and switching to a different literal in the same clause. An example of such a path for propositional calculus is the sequence $(\{p_{1},p_{2},p_{3}\},(p_{1},\neg p_{1}),\{\neg p_{1},q_{1},q_{2}\},(q_{1},\neg q_{1}),\{\neg q_{1},\neg r_{1}$, $\neg r_{2}\})$. The sequence $(\{p_{1},p_{2},p_{3}\},(p_{1},\neg p_{1}),\{\neg p_{1},q_{1},q_{2}\}$, $(\neg p_{1},p_{1}),\{p_{1},\neg r_{1},\neg r_{2}\})$ is not an alternating path.

Why this particular definition of alternating path is chosen will become clear later, as its properties are presented.

The length of an alternating path ($C_{1},\dots,C_{n}$) is $n$, counting only the clauses.

The relevance distance $d_{S}(C_{1},C_{2})$ of $C_{1}$ and $C_{2}$ in $S$ is the length of the shortest alternating path in $S$ from $C_{1}$ to $C_{2}$. This is a measure of how closely related to each other two clauses are. If there is no alternating path in $S$ from $C_{1}$ to $C_{2}$ then $d_{S}(C_{1},C_{2})$ is $\infty$.

If $T$ is a subset of set $S$ of clauses then $d_{S}(T,C)$ is $\min\{d_{S}(D,C):D\in T\}$. This is called the relevance distance of $C$ from $T$. Also, $R_{n,S}(T)=\{C\in S:d_{S}(T,C)\leq n\}$. This is frequently written as $R_{n}(T)$. Clauses in $R_{n,S}(T)$ are said to be relevant at distance n from $T$. If $d_{S}(T,C)<\infty$ then we say $C$ is relevant (for $S$ and $T$).

Two clauses are alternating connected or relevance connected in $S$ if there is an alternating path in $S$ between them.

Note that alternating connectedness is not transitive. Example: $C_{1}=\neg P\vee Q,C_{2}=\neg Q,C_{3}=Q\vee\neg R$, and $S=\{C_{1},C_{2},C_{3}\}$. $C_{1}$ and $C_{2}$ are alternating connected in $S$ as are $C_{2}$ and $C_{3}$, but $C_{1}$ and $C_{3}$ are not.

$Gr(S)$ is the set of ground instances of clauses in $S$. If $S$ has no constant symbols then one such symbol is added for purposes of making $Gr(S)$ non-empty.

Two clauses $C$ and $D$ are ground connected in $S$ if they have ground instances $C^{\prime}$ and $D^{\prime}$ that are relevance connected in $Gr(S)$.

The ground relevance distance $d^{g}(C_{1},C_{2})$ of $C_{1}$ and $C_{2}$ is $\min\{d_{Gr(S)}(C^{\prime}_{1},C^{\prime}_{2}):C^{\prime}_{1},C^{\prime}_{2}\mbox{ are ground}$ \mbox{ instances of C_{1}$and$C_{2}, respectively}\}.

Sometimes $d^{g}$ can be larger than $d$, for example, consider $C_{1}$ and $C_{3}$ where $C_{1}=p(a),C_{2}=\neg p(x)\vee p(f(x)),C_{3}=\neg p(f(f(x))$, and $S=\{C_{1},C_{2},C_{3}\}$.

A set $S$ of clauses is relevance connected if between any two clauses $C$, $D$ in $S$ there is an alternating path.

3 Properties of alternating path relevance

Theorem 3.1

If $S$ is a relevance connected set and $|S|=n$ then between any two clauses in $S$ there is an alternating path of length at most $2n-2$.

Proof

This proof is essentially from Plaisted and Yahya [PY03], with a slightly different notation. The idea is that if there is a relevance path between two clauses, then there is one in which each clause appears at most twice. Also, the clauses at the ends of the path clearly only need to appear once, or else there is a shorter path.

3.1 Minimal unsatisfiable sets of clauses

This section relates alternating paths to unsatisfiability of sets of clauses. This section and section 3.2 are basically reviews of known results, with some new formalism. First we have the following result [PY03] :

Theorem 3.2

If $S$ is a minimal unsatisfiable set of clauses then $S$ is relevance connected.

Using Theorem 3.1 above, we obtain the following:

Theorem 3.3

If $S$ is a set of clauses, $S^{\prime}$ is a minimal unsatisfiable subset of $S$, and $|S^{\prime}|=n$ then between any two clauses in $S^{\prime}$ there is an alternating path in $S^{\prime}$ of length at most $2n-2$.

Now, short proofs imply small minimal unsatisfiable sets of clauses, which in turn implies that if there is a short refutation from $S$ then there is a minimal unsatisfiable subset of $S$ in which any two clauses are connected by a short alternating path.

Theorem 3.4

If there is a resolution refutation of length $n$ from $S$ then there is a minimal unsatisfiable subset $T^{\prime}$ of $S$ in which any two clauses are connected by an alternating path of length at most $2n-2$.

Proof

Suppose there is a resolution refutation of length $n$ from $S$. Let $T$ be the set of input clauses (clauses in $S$) used in the refutation; then $|T|\leq n$. Also, $T$ is unsatisfiable so it has a minimal unsatisfiable subset $T^{\prime}$ with $|T^{\prime}|\leq n$. Then any two clauses in $T^{\prime}$ are connected by an alternating path of length at most $2n-2$.

We are not aware of any such result that applies to other relevance measures. If one wants to add a small proof restriction to other relevance measures, then one way to do this is to combine them with alternating path relevance.

The following result is also easily shown, but without a bound on the length of the path:

Theorem 3.5

If $S$ is a minimal unsatisfiable set of clauses then any two clauses in $S$ are ground connected.

Proof

The idea is that if $S$ is unsatisfiable then it has a finite unsatisfiable set of ground instances which therefore has a relevance connected subset including at least one instance of each clause in $S$.

3.2 Completeness and Set of Support

Using relevance, one can filter the potentially large set $S$ of input clauses (clauses in $S$) to obtain a smaller set $S^{\prime}$ of relevant clauses, and then one can search for a proof from $S^{\prime}$ instead $S$. This can be done using the concept of a set of support.

Definition 1

If $S$ is an unsatisfiable set of clauses, then a support set for $S$ is a subset $S^{\prime}$ of $S$ such that any unsatisfiable subset of $S$ has non-empty intersection with $S^{\prime}$.

Such support sets are easily constructed from interpretations of the input clauses in many cases. In particular, if $I$ is an interpretation of the set $S$ of clauses, then $\{C\in S:I\not\models C\}$ is a set of support for $S$. If it is decidable whether $I\models C$ for clauses $C$, then such a set of support can be effectively constructed. This is true, for example, for finite models of $S$.

Theorem 3.6

If $S$ is unsatisfiable, $S^{\prime}$ is a support set for $S$, there is a length $n$ refutation from $S$ and $m\geq 2n-2$ then $R_{m,S}(S^{\prime})$ is unsatisfiable.

This leads to the following theorem proving method $Rel_{m,S}(S^{\prime})$:

Choose $m$, compute $R_{m,S}(S^{\prime})$, and test $R_{m,S}(S^{\prime})$ for satisfiability.

If $|R_{m,S}(S^{\prime})|$ is much smaller than $|S|$, then $Rel_{m,S}(S^{\prime})$ can be much faster than looking for a refutation directly from $S$. However, because one does not know which $m$ to try, one can perform $Rel_{1,S}(S^{\prime})$, $Rel_{2,S}(S^{\prime})$, $Rel_{3,S}(S^{\prime})$ et cetera, interleaving the computations because even for a fixed $m$, $Rel_{m,S}(S^{\prime})$ may not terminate. This leads to the following theorem proving method:

for $i=1$ step 1 until infinity do in parallel $Rel_{i,S}(S^{\prime})$ od;

An example of a support set for $S$ mentioned above is the set of clauses contradicting an interpretation $I$ of $S$. This provides a way to use semantics ($I$) in theorem proving, which humans also commonly use. Also, if $T$ is a satisfiable subset of $S$, then $S\setminus T$ is a support set for $S$. For example, if $T$ is a collection of axioms from some satisfiable theory such as number theory then $S\setminus T$ is a support set for $S$. Typically one attempts to prove a theorem $R$ from some collection $A$ of general, satisfiable axioms. Then one converts $A\wedge\neg R$ to clause form, obtaining set $S_{1}\cup S_{2}$ of clauses where $S_{1}$ are the clauses coming from $A$ and $S_{2}$ are the clauses coming from $\neg R$. Then $S_{2}$ is a support set for $S_{1}\cup S_{2}$. Support sets are often specified with a theorem.

Definition 2

If $S$ is unsatisfiable and $U$ is a support set for $S$, then the $U$ support radius for $S$, $U_{S}^{rad}$, is the minimal $m$ such that $R_{m,S}(U)$ is unsatisfiable. If $S$ is satisfiable, then $m$ is $\infty$.

The $U$ support neighborhood for $S$, $U_{S}^{nbrhd}$, is then $R_{m,S}(U)$. A $U$ support neighborhood clause for $S$ is a clause in $U_{S}^{nbrhd}$ and a $U$ support neighborhood literal for $S$ is a literal appearing in a $U$ support neighborhood clause for $S$. The $U$ support diameter for $S$ is the maximum relevance distance between two clauses within the $U$ support neighborhood for $S$.

4 Time Bound to Compute Relevance

Relevance neighborhoods can be computed within a reasonable time bound, which makes it feasible to use relevance techniques in theorem proving applications involving large knowledge bases. The computation methods presented here are new. This computation requires finding all pairs of complementary unifiable literals in clauses of $S$, constructing a graph from them, and then applying breadth-first search to compute the relevance distances from a support set to all clauses in $S$.

4.1 Pairwise unification

To compute the time for the unifications, let $||S||$ be the length of $S$ in characters when written out as a string of symbols, and similarly $||C||$ and $||L||$ for clauses and literals, respectively. Suppose $L_{i}$, $1\leq i\leq n$ are the literals in $S$. Unification can be done in linear time [PW78], so testing all pairs of literals for unifiability takes time proportional to $\Sigma_{1\leq i,j\leq n}(||L_{i}||+||L_{j}||)$. In practice, term indexing [RSV01] permits this to be done much faster. However, with some algebra, $\Sigma_{1\leq i,j\leq n}(||L_{i}||+||L_{j}||)=2n\Sigma_{1\leq i\leq n}||L_{i}||$ which is quadratic in $||S||$.

4.2 The graph $G_{S}$

Let $G_{S}$ be a graph obtained from $S$ for purposes of computing relevance distances. The nodes $V$ of $G_{S}$ are triples $<L,C,in>$ and $<L,C,out>$ where $C\in S$ and $L\in C$. There are two kinds of edges in $G_{S}$: Type 1 edges from $<L,C,out>$ to $<M,D,in>$ for all nodes $<L,C,out>$ and $<M,D,in>$ in $V$ such that $L$ and $M$ are complementary unifiable. There are also type 2 edges from nodes $<L,C,in>$ to $<M,C,out>$ where $L$ and $M$ are distinct literals of $C$. Type 1 edges encode the links between clauses in an alternating path and type 2 edges encode switching from one literal of a clause to another literal in an alternating path. The number of edges can be quadratic in $||S||$.

A path in a graph is a sequence $(v_{1},v_{2},\dots,v_{n})$ in which there is an edge from $v_{i}$ to $v_{i+1}$ for all $i$. The length of this path is $n$. Then there is a direct correspondence between alternating paths in $S$ and paths in $G_{S}$. Suppose $C_{1},p_{1},C_{2},p_{2},\dots,C_{n-1},p_{n-1},C_{n}$ is an alternating path in $S$ and $p_{i}$ is the pair $(L_{i},M_{i+1})$ of literals. The corresponding path in $G_{S}$ is $<L_{1},C_{1},out>$, $<M_{2},C_{2},in>$, $<L_{2},C_{2},out>$, $<M_{3},C_{3},in>$, $\dots$, $<M_{n},C_{n},in>$. The length of the alternating path is $n$ but the length of the path in $G_{S}$ is $2(n-1)$ if $n>1$.

4.3 Relevance neighborhoods

Suppose one wants to find $R_{k}(U)$ where $U$ is a subset of $S$. This can be done using the well-known linear time breadth-first search algorithm [CLRS09], which outputs the distances of all nodes from the starting node (and can be easily modified to have more than one starting node). First one sets the distances of all nodes $<L,C,in>$ and $<L,C,out>$ for $C\in U$ to 1, and all other nodes have distances of infinity. Then one applies breadth-first search which computes the length of all shortest paths from the nodes $<L,C,out>$ for $C\in U$ to all other nodes. From this, $R_{k}(U)$ is obtained as the clauses $C$ appearing in nodes $<L,C,in>$ whose distance is less than or equal to $2(k-1)$. Because this graph has a size that is quadratic in $||S||$, the overall method is quadratic. However, to compute $R_{k}(U)$, it is only necessary to construct and search the portion of the graph consisting of nodes at distance $2(k-1)$ or less, which can result in a considerable savings of time, especially for very large knowledge bases and small $k$.

4.4 The propositional case

If $S$ is purely propositional then it is possible to find relevance neighborhoods much faster, as follows: The edges of type 1 are replaced by a smaller number of edges. Suppose $C_{1},C_{2},\dots,C_{m}$ are all the clauses in $S$ containing a positive literal $P$ and $D_{1},D_{2},\dots,D_{p}$ are all the clauses in $S$ containing $\neg P$. Then the type 1 edges from $<P,C_{i},out>$ to $<\neg P,D_{j},in>$ are replaced by edges from $<P,C_{i},out>$ to a new node $<P>$ and edges from $<P>$ to $<\neg P,D_{j},in>$. Also, simlar edges are added with the sign of $P$ reversed: the type 1 edges from $<\neg P,D_{j},out>$ to $<P,C_{i},in>$ are replaced by edges from $<\neg P,D_{j},out>$ to a new node $<\neg P>$ and edges from $<\neg P>$ to $<P,C_{i},in>$. This means that the path in $G_{S}$ corresponding to a path in $S$ becomes a little longer, but the number of edges in $G_{S}$ is reduced from quadratic to linear, making the whole algorithm linear time. Then an alternating path of length $n$ in $S$ corresponds to a path of length $3(n-1)$ in $G_{S}$ if $n>1$. For small relevance distances, one need not construct all of $G_{S}$, as before.

Bounds similar to these but less precise were given earlier [JP84, Pla80].

5 Branching Factor Argument

Now we give a new evidence that relevance can help to find proofs faster.

Suppose each (first-order) clause has at most $k$ literals and each predicate appears with a given sign in at most $b$ clauses in a set $S$ of clauses. Suppose a clause $C$ is in an alternating path; how many clauses $D$ can appear after it in alternating paths in $S$? If $C$ is the first clause in the path then any one of its up to $k$ literals can connect to at most $b$ other clauses, so there can be up to $bk$ clauses $D$ after $C$ in various alternating paths in $S$. If $C$ is not the first clause in the path, then it cannot exit by the same literal it entered by, so the number of clauses $D$ that can appear after it in various alternating paths in $S$ is at most $b(k-1)$. Thus there is a branching factor of at most $b(k-1)$ at each level except at the first level.

Suppose $S^{\prime}$ is a support set for $S$. Then there can be $|S^{\prime}|$ clauses $C_{1}$ that are the first clauses in some alternating path starting in $S^{\prime}$. There can be up to $bk$ clauses $C_{2}$ after $C_{1}$ in alternating paths in $S$ and for each clause $C_{i}$ for $i>1$ there can be at most $b(k-1)$ clauses after it in various alternating paths in $S$. So in paths $C_{1},C_{2},\dots,C_{n}$ starting in $S^{\prime}$ with various clauses $C_{1},\dots,C_{n}$, the number of clauses $C_{n}$ in all such paths is at most $|S^{\prime}|b^{n-1}k(k-1)^{n-2}$. All clauses in $R_{n}(S^{\prime})$ must appear in some such path. The total number of clauses appearing in such paths is then bounded by $|S^{\prime}|\Sigma_{1\leq i\leq n}b^{i-1}k(k-1)^{i-2}$. If $bk>1$ this is bounded by $2|S^{\prime}|b^{n-1}k(k-1)^{n-2}$. If this quantity is muich smaller than $|S|$ and $R_{n}(S^{\prime})$ is unsatisfiable then the effort to find a proof from $R_{n}(S^{\prime})$ can be much less than the effort to find a proof from all of $S$.

The alternating path approach is intended for proofs with small relevance bounds, so that the exponent $n$ should be small. Also, if clause splitting is used to break unifications, then the effective value of $b$ may be reduced. The value $k$ is typically small in first-order clause sets.

6 Experimental evidence with some knowledge bases

There is another evidence that relevance can help to reduce the size of the clause set that one must consider. This is based on an implementation of relevance [JP84] in which a first-order situation calculus knowledge base KB1 of about 200 clauses and a first-order knowledge base KB2 of about 3000 clauses expressing a map of a portion of the United States were considered and relevance methods were applied. For these examples, $R_{n}(S^{\prime})$ was computed for $S^{\prime}$ representing various queries and with increasing $n$ until a non-empty set of relevant clauses was found. This approach used additional pruning techniques to reduce the size of the relevant set such as a purity test in which clauses were deleted from $R_{n}(S^{\prime})$ if they had literals that did not complement unify with any other literals in $R_{n}(S^{\prime})$; additional details can be found in the paper.

[TABLE]

For query 5, the method instantiated the query itself and found four instances of the query, all needed for a refutation. In all cases a small set of unsatisfiable clauses was found. These results were pubished previously [JP84] but are not widely known.

7 Large Knowledge Bases

For some large theories, the actual refutations tend to be small; this also suggests that relevance techniques can be helpful. The following quote [PSST10] concerns the SUMO knowledge base:

”The Suggested Upper Merged Ontology (SUMO) has provided the TPTP problem library with problems that have large numbers of axioms, of which typically only a few are needed to prove any given conjecture.”

However, in this case, one is frequently testing to see if the axioms themselves are satisfiable. Because relevance techniques presented depend on knowing that a large subset of the axioms is satisfiable, in order to use relevance one would have to find such a subset even without knowing that all the axioms were satisfiable. The following quotation [RRG05] concerns another large knowledge base:

”…the knowledge in Cyc’s KB is common-sense knowledge. Common-sense knowledge is more often used in relatively shallow, ’needle in a haystack’ types of proofs than in deep mathematics style proofs.”

Concerning the Sine Qua Non approach [HV11], the authors write:

”Problems of this kind usually come either from knowledge-base reasoning over large ontologies (such as SUMO and CYC) or from reasoning over large mathematical libraries (such as MIZAR). Solving these problems usually involves reasoning in theories that contain thousands to millions of axioms, of which only a few are going to be used in proofs we are looking for.”

Of course, the knowledge base used for Watson [Fer12] was huge, but the proofs (if one can call them proofs) had to be found quickly, so they had to be relatively small.

8 Relationship to the Set of Support Strategy in Resolution

Another evidence that relevance can help comes from the usefulness in many cases of the set of support strategy [WRC65] for first-order theorem proving. This strategy is included as one of the standard options in many first-order theorem provers. This is because experience has shown that the set of support strategy often helps to find proofs faster. Some experimental evidence that first-order theorem proving strategies using set of support techniques outperform others for large theories has been obtained [RS98].

Now we show formally that the set of support strategy restricts attention to relevant clauses, and in fact, uses the most relevant clauses first. This result is new. It is surprising that set of support should correspond to relevance defined in terms of alternating paths in this way, because the definition of alternating paths is non-intuitive. Because the set of support strategy often helps to find proofs, this is evidence that relevance is also helpful for proof finding.

Definition 3

If $S$ is a set of first-order clauses then a resolution sequence from $S$ is a sequence $C_{1},C_{2},C_{3},\dots$ of clauses where each $C_{i}$ is either in $S$ or is a resolvent of $C_{j}$ and $C_{k}$ for $j,k<i$. There is a parent function that for any $i$ returns such a pair $(j,k)$, but mostly this will be left implicit.

Definition 4

Suppose $S$ is a set of first-order clauses and $S^{\prime}$ is a subset of $S$. Then a clause $C_{i}$ in a resolution sequence $C_{1},C_{2},C_{3},\dots$ from $S$ is $S^{\prime}$-supported if it is either in $S^{\prime}$ or at least one of its parents is $S^{\prime}$-supported in the sequence. The set of support strategy for $S$ with support set $S^{\prime}$ is the set of resolution sequences $C_{1},C_{2},\dots,C_{n},\dots$ from $S$ in which each non-input clause $C_{i}$ is $S^{\prime}$-supported in the resolution sequence.

The set of support strategy is complete in the sense that if $S$ is an unsatisfiable set of first-order clauses and $S^{\prime}$ is a support set for $S$ then there is a set of support refutation from $S$, that is, a resolution sequence from $S$ according to the set of support strategy for $S$ and $S^{\prime}$ in which $C_{n}$ is the empty clause, denoting false.

Definition 5

Suppose $P$ is the alternating path $C_{1},p_{1},C_{2},p_{2},\dots,C_{n-1},p_{n-1},C_{n}$ and $p_{i}=(L_{i}$, $M_{i+1})$ for $1\leq i<n$ and $n>1$. Let $C^{\prime}_{i}$ be $C_{i}\setminus\{L_{i},M_{i}\}$ for $1<i<n$. Let $C^{\prime}_{1}$ be $C_{1}\setminus\{L_{1}\}$ and let $C^{\prime}_{n}$ be $C_{n}\setminus\{M_{n}\}$. Then ${\cal P}^{*}$ is $C^{\prime}_{1}\cup C^{\prime}_{2}\cup\dots\cup C^{\prime}_{n}$. ${\cal P}^{*}$ is considered to be a clause denoting the disjunction of its literals. If $n=1$ then $(C_{1})^{*}=C_{1}$.

Definition 6

A collection ${\cal A}$ of alternating paths covers a first-order clause $C$ if for every literal $L\in C$ there is an alternating path $P\in{\cal A}$ and a literal $M\in P^{*}$ such that $L$ is an instance of $M$.

Theorem 8.1

Suppose ${\cal A}$ covers first-order clause $C$, and $D$ is a resolvent of $C$ and $C^{\prime}$ for some clause $C^{\prime}$. Let $L\in C$ and $L^{\prime}\in C^{\prime}$ be literals of resolution in the respective clauses. Let $P$ be a path in ${\cal A}$ and $L_{P}$ be a literal in $P^{*}$ such that $L$ is an instance of $L_{P}$. Let $P^{\prime}$ be a prefix of the path $P$ such that the last clause in $P^{\prime}$ contains $L_{P}$. Let $\Gamma$ be the path $P^{\prime}\circ((L_{P},L^{\prime}),C^{\prime})$, that is, $\Gamma$ is $P^{\prime}$ with $(L_{P},L^{\prime})$ and $C^{\prime}$ added to the end. Then $\Gamma$ is an alternating path and ${\cal A}\cup\{\Gamma\}$ covers $D$.

Proof

The fact that $\Gamma$ is an alternating path follows directly from the definitions. Let ${\cal A}^{\prime}$ be ${\cal A}\cup\{\Gamma\}$. Now, ${\cal A}^{\prime}$ covers $D$; all literals in $D$ that are instances of literals in $C^{\prime}$ are covered because $C^{\prime}$ is the last clause in $\Gamma$, and the literal $L^{\prime}$ which is possibly not in $\Gamma^{*}$ has been removed from $D$ by the resolution operation. The literals in $D$ that come from literals in $C$ are covered because the literals in $C$ were already covered by $P^{\prime}$ and the literals in $D$ have only been instantiated in the resolution operation.

Suppose $S$ is a set of first-order clauses and $S^{\prime}$ is a support set for $S$.

Theorem 8.2

Suppose $C_{1},C_{2},C_{3},\dots C_{n}$ is a resolution sequence in the set of support strategy for $S$ with support set $S^{\prime}$. Then for every literal $L$ in every derived (that is, non-input) clause $C_{i}$ there is an alternating path $P$ from $S^{\prime}$ of length at most $i$ such that $L$ is an instance of a literal in $P^{*}$; thus $L$ is an instance of a literal in a clause $D$ in $S$ at relevance distance at most $i$. Also, for every input clause $C_{i}$ in the proof there is an alternating path $P$ from $S^{\prime}$ of length at most $i$ ending in $C_{i}$.

This theorem is saying that there is an alternating path of length 1 from $S^{\prime}$ to $C_{1}$, an alternating path of length at most 2 from $S^{\prime}$ to $C_{2}$ if $C_{2}$ is an input clause, an alternating path of length at most 3 from $S^{\prime}$ to $C_{3}$ if $C_{3}$ is an input clause, and so on.

Proof

This follows from Theorem 8.1. By induction on $i$, showing this for $i$ = 1 and showing that if it’s true for $i$ it’s also true for $i+1$, there is a set ${\cal A}_{i}$ of alternating paths of length at most $i$ starting in $S^{\prime}$ and covering $C_{i}$ if $C_{i}$ is a derived clause. This implies all the conclusions of the theorem.

The implication of Theorem 8.2 for the set of support strategy is that all input clauses $C_{i}$ in the proof are relevant at distance at most $i$.

This shows that the set of support strategy restricts attention to relevant clauses, so that the effectiveness of this strategy is evidence that relevance is helpful. In fact, one can easily show that if $C$ is a clause in $S$ at relevance distance $n$ then $C$ appears in a set of support proof of length at most $2n-1$. Thus the set of support strategy uses exactly the relevant clauses in $S$.

8.1 Limitations of the Set of Support Strategy

The question now arises, if the set of support strategy is in some sense equivalent to relevance, then why not just use it all the time and dispense with relevance altogether?

First, there are some extensions to relevance that do not naturally incorporate into the set of support strategy; these involve a purity test and the use of multiple sets of support. These techniques have been presented [Pla80, JP84] in a couple of early papers.

Also, for propositional clause sets, $DPLL$ with CDCL [SLM09] is generally much better than resolution, so relevance techniques may have an advantage over resolution for such clause sets even with the set of support strategy.

In addition, for a Horn set, that is, a set of first-order Horn clauses, with an all-negative set of support, the set of support strategy is the same as input resolution, which requires every resolution to have one parent that is an input clause. Define the depth of a proof so that the depth of a (trivial) proof of an input clause is zero, and if $C$ is proved by resolving $C_{1}$ and $C_{2}$, then the depth of the proof of $C$ is one plus the maximum depths of the proofs of $C_{1}$ and $C_{2}$. Then the depth of a proof from a Horn set with an all-negative set of support is at least as large as the number of input clauses used, and could be larger if more than one instance of an input clause is used. This implies that it is possible to have a clause set with a small support radius but which requires a large number of input clauses and therefore a very deep set of support proof. Thinking in terms of Prolog style subgoal trees, the relevance distance corresponds roughly to the depth of the tree but the length of a set of support proof corresponds to the number of nodes in the tree. For such clause sets, it may be better to use hyper-resolution, which basically resolves away all negative literals of a clause at once, because the proof depth corresponds only to the depth of the tree.

As an example, consider the propositional clause set $\neg p,p\vee\neg q_{1}\vee\neg q_{2}\vee\neg q_{3},q_{1}\vee\neg r_{1}\vee\neg r_{2},q_{2}\vee\neg r_{3}\vee\neg r_{4},q_{3}\vee\neg r_{5}\vee\neg r_{6}$ together with the unit clauses $r_{1},r_{2},r_{3},r_{4},r_{5},r_{6}$. Suppose $\neg p$ is the set of support. Then the set of support strategy resolves $\neg p$ with $p\vee\neg q_{1}\vee\neg q_{2}\vee\neg q_{3}$ to produce $\neg q_{1}\vee\neg q_{2}\vee\neg q_{3}$. This then resolves with $q_{1}\vee\neg r_{1}\vee\neg r_{2}$ to produce $\neg r_{1}\vee\neg r_{2}\vee\neg q_{2}\vee\neg q_{3}$. Two more resolutions produce $\neg r_{1}\vee\neg r_{2}\vee\dots\vee\neg r_{6}$ and six more resolutions with unit clauses produce a contradiction, for a total of ten resolutions. However, all clauses are within a relevance distance of three of the set of support. Also, hyper-resolution can find a refutation in fewer levels of resolution. First, the unit clauses hyper-resolve with $q_{1}\vee\neg r_{1}\vee\neg r_{2},q_{2}\vee\neg r_{3}\vee\neg r_{4},q_{3}\vee\neg r_{5}\vee\neg r_{6}$ to produce the units $q_{1},q_{2},q_{3}$ and these hyper-resolve with $p\vee\neg q_{1}\vee\neg q_{2}\vee\neg q_{3}$ to produce the unit $p$ which then resolves with $\neg p$ to produce a contradiction. If the subgoal tree is larger the difference can be more dramatic; the number of levels of resolution by the set of support strategy can be exponentially larger than the number of levels of hyper-resolution required. If there are many clauses in the clause set, a deep proof can easily get lost in the huge search space that is generated. The point is, although set of support restricts attention to relevant clauses, it does not always process them in the most efficient way, so it may be better to separate relevance detection from inference in some cases. The set of support strategy intermingles a relevance restriction with resolution inference.

9 DPLL

This section presents an extension of alternating path relevance to the $DPLL$ method. The $DPLL$ method for testing satisfiability of propositional clause sets is now of major importance in many areas of computer science, especially with the extension to CDCL [SLM09]. For example, an open conjecture concerning Pythagorean triples was recently solved in this way [HKM16]. Therefore any improvement in $DPLL$ is of major importance. Relevance can decrease the work required for $DPLL$ if the number of $U$ support neighborhood literals is small for a support set $U$. This decrease in work requires a modification to $DPLL$.

The $DPLL$ method, without unit rules or CDCL, can be expressed this way, for $Q$ a set of propositional clauses:

[TABLE]

Here $Q|L$ is $Q$ with all clauses containing $L$ deleted and $\neg L$ removed from all clauses containing $\neg L$. Also, $Q|\neg L$ is defined similarly with the signs of the literals reversed. We say that $L$ is the split literal or the literal chosen for splitting, in this case.

$DPLL$ also has unit rules. Essentially, if $Q$ contains a unit clause $L$ then $L$ is used to simplify $Q$ by replacing all occurrences of $L$ by true and simplifying, and replacing all occurrences of $\neg L$ by false and simplifying. There is a similar rule if there is a unit clause $\neg L$ in $Q$, with signs reversed.

9.1 Relevant $DPLL$

The material in this section is new. Let $U$ be a support set for a set $S$ of propositional clauses. Define the relevance distance $d(U,L)$ of a literal $L$ from $U$ to be the minimal $n$ such that $L$ or its negation appears in a clause $C$ of relevance distance $n$ from $U$.

Definition 7

If $S$ is a set of clauses then ${\cal L}(S)$ is $\bigcup S$, that is, the set of literals in clauses of $S$.

Definition 8

The stepping sequence for a set $S$ of propositional clauses and a support set $U$ for $S$ is a sequence $({\cal L}_{1},{\cal L}_{2},\dots,{\cal L}_{n})$ where ${\cal L}_{i}$ is the set of literals of $S$ at relevance distance $i$ from $U$, and $n$ is the maximum relevance distance from $U$ of any literal in $S$. A stepping remainder sequence for $S$ and $U$ is a sequence $({\cal L}^{\prime}_{1},{\cal L}^{\prime}_{2},\dots,{\cal L}^{\prime}_{n})$ such that for the stepping sequence $({\cal L}_{1},{\cal L}_{2},\dots,{\cal L}_{n})$ for $S$ and $U$, ${\cal L}^{\prime}_{i}\subseteq{\cal L}_{i}$ for all $i$. A leading literal of a stepping remainder sequence $({\cal L}^{\prime}_{1},{\cal L}^{\prime}_{2},\dots,{\cal L}^{\prime}_{n})$ is an element of ${\cal L}^{\prime}_{i}$ or the complement of such an element, where $i$ is minimal such that ${\cal L}^{\prime}_{i}$ is non-empty. Also, $Step:T$ where $T$ is a set of clauses and $Step=({\cal L}^{\prime}_{1},\dots,\dots,{\cal L}^{\prime}_{n})$ denotes the stepping remainder sequence $({\cal L}^{\prime\prime}_{1},{\cal L}^{\prime\prime}_{2},\dots,{\cal L}^{\prime\prime}_{n})$ in which ${\cal L}^{\prime\prime}_{j}={\cal L}^{\prime}_{j}\cap{\cal L}(T)$ for all $j$. If $Step$ is the stepping remainder sequence $({\cal L}_{1},{\cal L}_{2},\dots,{\cal L}_{n})$ then $|Step|$ is $\Sigma_{i}|{\cal L}_{i}|$.

Note that support sets are easy to obtain for propositional clause sets. For example, the set of all-positive clauses and the set of all-negative clauses are both support sets for all propositional clause sets.

The *relevant * $DPLL$ method $DPLL$-$Rel$ can be expressed this way, where $Q$ is a set of propositional clauses and $StepR$ is a stepping remainder sequence for $Q$:

[TABLE]

This procedure is called at the top level as $DPLL$-$Rel(S,Step)$ with $Step$ as the stepping sequence for $S$ and $U$, and $U$ as a support set for $S$. If there is a choice of leading literals, then any DPLL heuristic can be used to choose among them. Now, the recursive calls to $DPLL$-$Rel$ will have stepping sequences with fewer literals; that is, $|StepR|$ will be smaller with each level of recursion. This enables proofs of properties of $DPLL$-$Rel$ by induction. Also, for the recursive calls to $DPLL$-$Rel(Q,StepR)$, $StepR$ will always include all $U$ support neighborhood literals of $S$ that remain in $Q$, so that if $|StepR|=0$ then $Q$ contains no $U$ support neighborhood literals of $S$. If at this point $Q$ does not contain the empty clause, then one who understands $DPLL$ can see that this means that $DPLL$-$Rel$ has essentially found a model of the relevant clauses of $S$. By theorem 3.2, $S$ is satisfiable.

These considerations justify returning ”sat” if $StepR_{i}$ is empty for all $i$ in line 2 of $DPLL$-$Rel$. Then a model of the relevant clauses can be returned, without even exploring the remaining literals. However, this depends on $U$ being a valid support set for $S$, that is, $S\setminus U$ is satisfiable. If there is some doubt about this, then line 2 of $DPLL$-$Rel$ should be replaced by the following:

[TABLE]

This means that ordinary $DPLL$ is called in this case to explore the remaining literals.

Theorem 9.1

If $S$ is an unsatisfiable set of propositional clauses, $Step$ is the stepping sequence for $S$ and a support set $U$ for $S$, and $DPLL$-$Rel(S,Step)$ is called at the top level, then the number of recursive calls to $DPLL$-$Rel(Q,StepR)$ for various stepping remainder sequences $StepR$ is bounded by $2^{k}$ where $k$ is the number of literals in the $U$ support neighborhood for $S$.

Proof

Let $m$ be the $U$ support radius for $S$. If $StepR=({\cal L}_{1},{\cal L}_{2},\dots,{\cal L}_{n})$ is a stepping remainder sequence for $S$, let $StepR^{j}$ be $({\cal L}_{1},{\cal L}_{2},\dots,{\cal L}_{j})$. By induction on $|StepR|$ for the recursive calls, one shows that for every call to $DPLL$-$Rel(Q,StepR)$, the set of clauses in $Q$ over the $U$ support neighborhood literals in $StepR^{m}$ is unsatisfiable. The proof makes use of the lemma that if $Q$ is unsatisfiable so are $Q|L$ and $Q|\neg L$ for any literal $L$. If $|StepR^{m}|=0$ then $R$ in the recursive call must contain the empty clause. Each recursive call to $DPLL$-$Rel$ reduces the value of $|StepR^{m}|$ by one or more, and there are at most two recursive calls to $DPLL$-$Rel$, whence the $2^{k}$ bound follows.

Note that $k$ can be much smaller than the set of all literals in relevant clauses in $S$. This result is not trivial, because some clauses in $Q$ are deleted in $Q|L$ and $Q|\neg L$ and even clauses that have literals that are not in $StepR^{m}$ can contribute to such deletions by a succession of unit deletions in $DPLL$. Therefore one needs to show that the clauses over the literals in $StepR^{m}$ are still unsatisfiable in $Q|L$ and $Q|\neg L$.

This bound of $2^{k}$ can be much smaller than the worst case $2^{n}$ bound on ordinary $DPLL$ where $n$ is the number of atoms (predicate symbols) in $S$. It is not necessary to know the $U$ support neighborhood in order to apply this method. This method can be extended to CDCL, as before. Of course, in practice the number of calls is likely to be much smaller than $2^{k}$. There are already heuristics for DPLL, but to our knowledge none of them decreases the worst case time bound as this approach does.

If the $DPLL$ unit rules are used freely, units that are not in the $U$ support neighborhood may be processed by the unit rules even if the split literals are handled as in $DPLL$-$Rel$. The choices to deal with this are to allow all units to be used in the unit rules, or to restrict the unit rules to only use units that are relevant in some sense.

10 Other extensions of alternating path relevance

10.1 Splitting clauses

Another idea that can help to make relevance more effective is to split a clause into several clauses that together have the same ground instances. For example, a clause $C[x]$ containing an occurrence of a variable $x$ can be split into $C[f_{1}(\overline{y^{1}})],C[f_{2}(\overline{y^{2}})],\dots$, $C[f_{n}(\overline{y^{n}})]$ where $f_{1},f_{2},\dots,f_{n}$ are all the function symbols appearing in the clause set and $\overline{y^{i}}$ are sequences of new variables. If the clause set has no constant symbols, then one such symbol has to be allowed, in addition. This idea can be extended in the obvious way to clauses containing more than one variable. This idea can be especially hepful with general axioms such as the equality axioms. For example, the axiom $x=y\rightarrow y=x$ can join many clauses together, causing all clauses to be at small relevance distances from one another. This can hinder the application of relevance in systems involving equality. Splitting clauses that have literals unifying with the complements of literals in many other clauses can help a lot in such cases. Also, it can be helpful to choose which variable to split so that the largest number of complement unifications from literals in the clause is broken. Spltting clauses was one of the techniques used earlier [JP84]; the technique used there was to split a clause $C[x]$ into two clauses $C_{1}[x]$ and $C_{2}[x]$ where $C_{1}$ restricts $x$ to be instantiated to one of $f_{1}(\overline{y^{1}})\cdots f_{k}(\overline{y^{k}})$ for some $k$ and $C_{2}$ restricts instantiation of $x$ to one of $f_{k+1}(\overline{y^{k+1}})\cdots f_{n}(\overline{y^{n}})$.

10.2 A fine type theory

Also, a very fine type theory can help with equality and with relevance in general. Types can be incorporated into the unification algorithm, so that for example a variable of type ”person” would not unify with a variable of type ”building.” This idea can increase the relevance distance between clauses and make relevance more effective.

10.3 Multiple sets of support

If one has a finite collection $\{I_{1},I_{2},\dots,I_{n}\}$ of interpretations of a set $S$ of clauses, then let $T_{i}$ be $\{C\in S:I_{i}\not\models C\}$. Then the $T_{i}$ are sets of support for all such $i$. Let $R^{*}_{n,S}$ be $R_{n,S}(T_{1})\cap R_{n,S}(T_{2})\cap\cdots R_{n,S}(T_{k})$. Then if $S$ is unsatisfiable, it is easy to see that $R^{*}_{n,S}$ will also be unsatisfiable for some $n$. The method of Jefferson and Plaisted [JP84] was basically to compute $R^{*}_{n,S}$ for various $n$ and apply a purity test until a non-empty set was obtained. Also, clause splitting was used. These techniques proved to be effective for the problems tried.

11 Equality

Equality can cause a problem for relevance if literals of the form $s=t$ or their negations appear in many clauses. It may be acceptable to use the equality axioms without any special modification for equality. However, there are also other possibilities. In general, there needs to be thought devoted to how to integrate equality and relevance, possibly using some kind of completion procedure [BN98] combined with relevance.

Ordered paramodulation is a theorem proving technique that is effective for clause form resolution with equality. It basically uses equations, replacing the large (complex) side of the equation by the small (simple) side. However, it is not compatible with the set of support strategy. For example, consider this set: $f(a,x)=x$, $f(y,b)=y$, $a\neq b$. If $a\neq b$ is the set of support, then a refutation requires using the equations in the wrong direction for ordered paramodulation and even paramodulating from variables. This kind of paramodulation can be highly inefficient compared to ordered paramodulation. This is another evidence that just applying the set of support strategy is not always the best way to handle relevance.

For equality and relevance there are at least several choices: 1. Find the relevant clauses by some method, then use a strategy such as ordered paramodulation and hyper-resolution to find the proof. 2. Use Brand’s modification method [Bra75] on the set of clauses, find the relevant set of clauses, and then apply some inference method to find the proof. 3. Modify relevance to take into account equality, possibly by using a set of equations to rewrite things before computing relevance or by incorporating E-unification [BS01] into the unification algorithm.

12 Discussion

The features of alternating path relevance have been reviewed, and some extensions including an extension to $DPLL$ have been presented. Graph based methods for computing this relevance measure have been presented. Alternating path relevance has an unexpected relationship with the set of support strategy. Some previous successes with it as well as with the set of support strategy argue that this method can be effective. A couple of theoretical results indicating the effectiveness of alternating path relevance have also been presented. Evidence has been given that several large knowledge bases frequently permit small proofs, suggesting that alternating path relevance and other relevance methods can be effective for them. An open problem for many of the relevance techniques is to give a theoretical justification for their effectiveness.