Exploiting Instantiations from Paramodulation Proofs in Isabelle/HOL
Lukas Bartl, Jasmin Blanchette, Tobias Nipkow
TL;DR
This paper introduces a new tool that analyzes Metis proofs in Isabelle/HOL to derive variable instantiations, enhancing proof automation, speed, and user understanding.
Contribution
The novel tool automatically extracts variable instantiations from Metis proofs, improving Sledgehammer's effectiveness and proof transparency in Isabelle/HOL.
Findings
Increases Sledgehammer's success rate
Speeds up proof generation
Aids user understanding of proof steps
Abstract
Metis is an ordered paramodulation prover built into the Isabelle/HOL proof assistant. It attempts to close the current goal using a given list of lemmas. Typically these lemmas are found by Sledgehammer, a tool that integrates external automatic provers. We present a new tool that analyzes successful Metis proofs to derive variable instantiations. These increase Sledgehammer's success rate, improve the speed of Sledgehammer-generated proofs, and help users understand why a goal follows from the lemmas.
| One-line proofs | |||||
| Evaluation run | W/o inst. | With inst. | Isar proofs | Failed | Total |
| 1 (instantiate = smart) | |||||
| 2 (instantiate = true) | |||||
| Evaluation run | Sledgehammer | Generated metis proof |
| 1 (instantiate = smart) | ms | ms |
| 2 (instantiate = true) | ms | ms |
| Evaluation run | Standard metis | With options | With ext | Total |
| 1 (instantiate = smart) | ||||
| 2 (instantiate = true) |
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
\excludeversion
rep \includeversionconf
\includeversionold-argcong \excludeversionnew-argcong
{conf}{rep}11institutetext: Universität Augsburg, Augsburg, Germany
11email: [email protected] 22institutetext: Ludwig-Maximilians-Universität München, Munich, Germany
22email: [email protected] 33institutetext: Technische Universität München, Munich, Germany
33email: [email protected]
Exploiting Instantiations from Paramodulation Proofs in Isabelle/HOL
Lukas Bartl(🖂) 11
Jasmin Blanchette 22
Tobias Nipkow 33
Exploiting Instantiations from Paramodulation Proofs in Isabelle/HOL (Technical Report)
Lukas Bartl(🖂) 11
Jasmin Blanchette 22
Tobias Nipkow 33
Abstract
Metis is an ordered paramodulation prover built into the Isabelle/HOL proof assistant. It attempts to close the current goal using a given list of lemmas. Typically these lemmas are found by Sledgehammer, a tool that integrates external automatic provers. We present a new tool that analyzes successful Metis proofs to derive variable instantiations. These increase Sledgehammer’s success rate, improve the speed of Sledgehammer-generated proofs, and help users understand why a goal follows from the lemmas.
Keywords:
instantiations paramodulation Isabelle/HOL Sledgehammer
1 Introduction
Sledgehammer [31] is undoubtedly one of the interactive proof assistant Isabelle/HOL’s [27] most popular components. This component gives Isabelle access to many external automatic theorem provers (ATPs), which can mechanize routine proofs [12, 16]. When the user invokes Sledgehammer on a proposition to be proved, it performs the following steps:
It invokes a number of external ATPs with the goal and the background library of facts (definitions, lemmas, theorems, etc.); this involves translating the goal and facts from higher-order logic to the ATPs’ logics. 2. 2.
For any proof that is found, the used facts are extracted from it and the internal prover Metis [20], based on ordered paramodulation, is invoked to reconstruct an Isabelle proof based on this list of facts only.
This paper is about obtaining and exploiting the instantiations of the facts used in the proofs. Our motivation is that using instantiated facts
- –
can speed up proof reconstruction (and even turn timeouts into successes);
- –
can improve readability of proofs for humans; and
- –
can lead to simpler proofs.
Example 1
We start with a simple example where we believe human readability is improved. We work in a linear order with and . Sledgehammer proves that in the context of the facts
[TABLE]
the assumption implies , where and are arbitrary constants. Why is that? It turns out that the following instantiations are used:
[TABLE]
Clearly implies via . This implies and via and . Now we obtain and via and , and thus . Although this deduction still needs some thought, we believe that it is far simpler than if we had to work out the derivation from and alone.
This paper is structured as follows. Section 2 presents the necessary background about Isabelle, Sledgehammer, and Metis. Section 3 presents further introductory examples. The core of the paper is in Section 4, which explains how to obtain the instantiations used during a Metis proof and translate them into instantiations of the corresponding Isabelle facts. Section 5 explains how Sledgehammer was extended to cooperate with the extended Metis prover. Section 6 presents our empirical evaluation of how instantiations improve the performance of Sledgehammer. We end with related work and the conclusion.
Our extensions are available as part of Isabelle starting with version 2025.111https://isabelle.in.tum.de/website-Isabelle2025/ They are documented in the Sledgehammer user’s manual [5]. The raw data for our evaluation is available online.222https://nekoka-project.github.io/pubs/instantiations˙data.zip
2 Background
{rep}
Our work is based on Isabelle (Section 2). Particularly relevant are its Sledgehammer advisory tool (Section 2) and its metis proof method (Section 2), which relies on the Metis ATP (Section 2).
Isabelle.
Isabelle/HOL [41, 27] is a proof assistant for polymorphic higher-order logic enriched with type classes [34]. It is written primarily in Standard ML. It has an inference kernel through which all logical inferences must go to be deemed acceptable. At the user level, notations largely follow mathematical practice.
Sledgehammer.
The Sledgehammer tool [31] consists of six main components:
The relevance filter (or “premise selector”) heuristically selects a subset of the available facts as likely relevant to the current goal. Typically, hundreds of facts can be chosen without overwhelming ATPs. Sledgehammer includes two relevance filters [23, 8], which can be combined. 2. 2.
The translation module constructs an ATP problem from the selected facts and the current goal, translating Isabelle’s polymorphic higher-order logic to the ATP’s logic [22, 7]. 3. 3.
The ATP tries to prove the problem. Actually, multiple ATPs can be run in parallel. Commonly used ATPs include cvc5 [3], E [39], Leo-III [35], SPASS [10], Vampire [4], veriT [13], Z3 [25], and Zipperposition [38]. 4. 4.
If one or more ATPs find a proof, the proof minimization module repeatedly invokes each ATP with subsets of the facts referenced in the respective proof, trying to reduce the number of dependencies and speed up the next steps. 5. 5.
The proof reconstruction module transforms each ATP proof into a textual Isabelle proof. Reconstruction means that Sledgehammer and the ATP need not be trusted. Typically, the structure of the ATP proof is discarded, and the Isabelle proof consists of a single proof method (often metis) invoked with the facts referenced in the ATP proof. Detailed Isabelle proofs, or Isar proofs [40], are available as an experimental feature [6]. 6. 6.
For each ATP proof, the preplay module tries out various proof methods before they are presented to the user. If several methods succeed, the fastest one is chosen [6].
As an example, suppose Sledgehammer selects 512 facts and passes them, along with the goal, to E. Then E finds a proof involving three facts, , , and , and minimization reduces this list to two: and . By trial and error (and preplaying), Sledgehammer determines that the proof method metis with and as arguments solves the Isabelle goal in ms, and no other proof method succeeds, so this metis call is suggested to the user.
The Metis ATP.
Metis [20] is an ATP for untyped first-order logic with equality written in Standard ML. It is based on ordered paramodulation, a variant of superposition [2]. Although Metis was developed as a standalone program, it is also incorporated in Isabelle’s source code, so that it is always available.
Thanks to its calculus, Metis is reasonably performant, although it cannot compete with state-of-the-art superposition provers such as Vampire [36]. Metis’s main strengths are the readability of its source code and the simplicity and fine granularity of its proof format. Proofs are expressed using the following six inference rules:
{prooftree} \hypoC \infer1[Axiom] C {prooftree} \hypoC \infer1[Assume] A ∨¬A {prooftree} \hypoC \infer1[Subst] Cσ
{prooftree} \hypoC \infer1[Refl] t = t {prooftree} \hypoC \infer1[Equality] s ≠t ∨¬L[s]_p ∨L[t]_p {prooftree} \hypoC ∨A \hypo¬A ∨D \infer2[Resolve] C ∨D
Notice that substitution is captured by the explicit Subst rule instead of being part of Resolve. Moreover, all equality reasoning is reduced to Refl and Equality. For proof search, Metis relies on a more efficient calculus that performs ordered paramodulation, but the proofs are then translated to the above fine-granular rules.
The metis Proof Method.
Isabelle’s metis (with a lowercase m) proof method [32] builds on the Metis (with an uppercase M) ATP to provide general-purpose proof automation. The metis proof method takes a list of facts as argument and translates them, together with the current goal, from polymorphic higher-order logic to untyped first-order logic. The resulting axiom clauses are then introduced in the proof attempt using the Axiom rule. The translation uses the same techniques (and the same code) as Sledgehammer. Next, metis invokes the Metis ATP, and if Metis finds a proof, it is reconstructed step by step using Isabelle’s inference kernel, so that the goal becomes an Isabelle theorem. A metis call is considered successful if the Metis ATP found a proof and Isabelle reconstructed it.
Since it may be hard for the user to determine which facts are necessary for a proof, in practice metis is almost always used in conjunction with Sledgehammer.
3 Examples
Before we study our metis and Sledgehammer extensions in detail, we take a look at a few examples that illustrate how instantiating facts can not only improve readability but also speed up proof reconstruction and lead to simpler proofs. The final example demonstrates how more complex terms are displayed.
Example 2
We present a goal from the Archive of Formal Proofs [9], where instantiating facts substantially speeds up the proof and turns a timeout into a success. This example stems from a formalization of Tarski’s axioms for Euclidean geometry [15]. The lemma cong_mid2__cong states the following property, where denotes the congruence relation and states that a point lies exactly in the middle of two other points:
[TABLE]
When invoking Sledgehammer on a modern laptop for this goal, the external ATP cvc5 [3] finds a proof using the facts cong_inner_transitivity, l2_l1_b, midpoint_bet, and midpoint_cong. However, metis times out; i.e., it fails to find a proof derived from these facts within s. After inferring the instantiations and instantiating the facts, though, metis successfully solves the goal in ms. The instantiations are simple: Each free variable in the facts is replaced by one of . The facts midpoint_bet and midpoint_cong each have two different instantiations and thus appear twice in the resulting Isabelle proof, which may also improve readability.
Example 3
Instantiating facts not only speeds up metis but also leads to simpler and more readable proofs. Suppose that we want to prove the goal for natural numbers . When invoking Sledgehammer on a modern laptop, the external ATP veriT [13] finds a proof including the following facts, where returns the successor of its argument:
[TABLE]
The metis method can solve the goal with these facts in ms. However, after inferring the instantiations and instantiating the facts, metis needs only ms to solve the goal, and the auto [29] proof method is even faster, needing only ms. Additionally, auto requires only the following two instantiated facts:
[TABLE]
By contrast, auto fails to find a proof derived from these facts without instantiations. The Isabelle proof using auto is simpler and more readable than the metis proof, since trivial facts were eliminated. This helps the user focus on the relevant facts when trying to understand the proof.
Example 4
In the preceding examples, the instantiations were simple, with each free variable being replaced by a single symbol. There are also proofs where instantiations are more complex, possibly involving -abstractions and quantified variables. Consider the goal
[TABLE]
where states that a function is surjective. We have the fact at our disposal. Our metis extension suggests instantiating with and with in the fact. Notice that bound variables can be renamed in the inferred instantiations (here, became ). Additionally, ‘’—which corresponds to a fresh free variable—can appear as a placeholder for quantified variables (e.g., ), since there is no way to refer to them.
4 Instantiations from Metis Proofs
We extend the metis proof method so that it infers the instantiations of facts used in a proof and suggests them to the user. This extension is executed after a successful metis proof, provided that the metis_instantiate option is enabled.
Inference of Instantiations for Metis Proofs.
To infer the instantiations of the Isabelle facts, we first infer the instantiations of the Metis clauses. These are the substitutions that were applied to the axiom clauses in the Metis proof.
Definition 1
A Metis theorem is recursively defined as a triple consisting of a Metis clause , an inference rule , and a list of Metis theorems , where is derived directly from using the rule . A Metis proof is a Metis theorem for the empty clause, denoted by .
Definition 2
A Metis substitution is a partial function from Metis variable names to Metis terms. The composition of two substitutions first applies and then , so that .
Definition 3
The function returns a list of pairs , where is an axiom clause in the Metis theorem and is the applied substitution. The argument is used as an accumulator and is initialized to for a Metis proof. The function is defined recursively as follows:
[TABLE]
where in the second equation is the substitution carried by the Subst inference rule on the left, and in the third equation denotes list concatenation.
Example 5
Suppose that we want to prove the goal on natural numbers from the following facts:
[TABLE]
We translate the goal and the facts into first-order logic by using the predicate symbol for . Metis finds the following proof:
- (1)
Axiom: 2. (2)
Axiom: 3. (3)
Subst from (2) using :
4. (4)
Axiom: 5. (5)
Equality: 6. (6)
Resolve from (4) and (5): 7. (7)
Resolve from (3) and (6): 8. (8)
Subst from (7) using :
9. (9)
Resolve from (1) and (8): 10. (10)
Axiom: 11. (11)
Subst from (10) using : 12. (12)
Resolve from (9) and (11):
We can now infer the substitutions that were applied to the axiom clauses:
[TABLE]
Given a Metis proof , the call yields a list of pairs , where is an axiom clause and is the applied substitution. If the proof is retried, the instantiated axiom clauses—i.e., the list of all clauses corresponding to the pairs —can be used instead of the original axiom clauses to restrict the search space and speed up Metis. After instantiation, a Metis proof is still possible:
Theorem 4.1
A Metis proof can be transformed into a new Metis proof derived from the instantiated axiom clauses (defined above). The new Metis proof does not involve the Subst inference rule and uses at most as many proof steps as .
The proof is by induction on the structure of the Metis proof . It proceeds by instantiating the clauses in with ’s current accumulator value and removing Subst inference steps. Because clauses are sets and not multisets of literals, instantiation can shorten clauses by unifying literals. This can result in the elimination of resolution steps and thereby entire subproofs.
Proof
We annotate each Metis theorem in with ’s current accumulator value using the call , defined as follows:
[TABLE]
where is again the substitution carried by the Subst inference rule.
It suffices to show that each annotated Metis theorem contained in can be transformed into a new Metis theorem such that holds (with for axiom clauses), Subst inference steps are removed, and no new proof step is introduced. Since implies , the whole Metis proof is transformed into a new Metis proof. By the definitions of the and functions, for axiom clauses implies that the new Metis proof is derived from the instantiated axiom clauses.
Assume that we have an annotated Metis theorem contained in . We prove the proposition by induction on the inference rule . In the base case, has no premises, so and is one of Axiom, Assume, Refl, and Equality. We construct the new Metis theorem so that holds.
If , we have , where , and , where is the substitution carried by the Subst inference rule. Inductively, we can transform into a new Metis theorem with . Using , we can deduce , so we use as the new Metis theorem and remove the substitution step.
If , then , where , , , , and . Inductively, we can transform into a new Metis theorem with and into with . Since instantiation can shorten clauses, we distinguish between two cases: In the first case, is contained in the clauses and , so we obtain and , where and . Then we can construct the new Metis theorem and holds since . In the second case, we find such that does not contain . Then holds, and we can use as the new Metis theorem, thereby removing the resolution step as well as the other subproof.
Translation of Metis Terms to Isabelle Terms.
Once the instantiations of the Metis clauses have been inferred, a translation procedure converts the contained Metis terms into Isabelle terms that can be presented to the user.
The list obtained from the function is filtered to include only those pairs where the Metis clause is derived from an Isabelle fact , and not from the negated goal. The procedure then continues to translate only Metis terms of substitution elements where the Metis variable occurs in and corresponds to a free variable in . There may be other Metis variables, including those corresponding to type variables or quantified variables. The direct instantiation of type variables is seldom useful, since types can be inferred from the instantiated terms. Metis variables that correspond to quantified Isabelle variables may emerge during clausification; however, since these are bound in Isabelle, there is no way to refer to them.
The translation procedure must decode constructs that are introduced by the translation from polymorphic higher-order logic to untyped first-order logic in metis. This decoding uses the same code as the proof reconstruction in metis [32], where Metis terms emerge during reconstruction of the Subst inference rule. The introduced constructs include the encoding of Isabelle symbols using Metis constants, the encoding of free variables using Metis variables, and the encoding of partial application using a distinguished binary symbol [22]. For example, and might be translated to and , where and are Metis constants and is a Metis variable. While decoding these constructs is straightforward, decoding the type information (for which there are several encodings [7]) was considered too complicated for proof reconstruction in metis, so type inference is used instead [32].
The resulting Isabelle terms may still contain Skolem terms and encodings of -abstractions. Such metis-internal constructs may not appear in terms suggested to the user. Thus our translation procedure must eliminate them.
Skolem terms are eliminated by replacing them with a wildcard, which is displayed as ‘’ and corresponds to a fresh free variable (as demonstrated in Example 4). Consequently, the instantiated facts may still contain free variables, for which Metis must substitute the corresponding Skolem terms. It does not suffice to simply replace the Skolem symbols with ‘’; their arguments must also be removed. This is necessary because the arity of Skolem symbols can change due to the instantiation of facts. For example, the Skolem symbol introduced for depends on and therefore requires an argument, whereas the Skolem symbol for the instance does not require any argument.
The metis proof method encodes -abstractions using either combinators [37] or -lifted supercombinators [19]. To eliminate these combinators, the translation procedure replaces them with their definition, followed by a -reduction of the resulting terms. For example, the term is encoded as , where . When the translation procedure detects the term , it replaces with and produces the term [math] after -reduction. If -abstractions remain after -reduction, the bound variables may have different names than in the original -abstractions (as demonstrated in Examples 4 and 7), but this is not a problem since Isabelle equates terms up to -equivalence.
In the final step of the translation procedure, all remaining free variables (i.e., all free variables occurring in the Isabelle terms and all uninstantiated variables in the clauses) are instantiated with the Isabelle polymorphic constant , which provides a witness for every type’s inhabitedness. These variables emerge when the proof was possible without concrete terms. For example, can be derived from without choosing a value for . Instantiation of such variables is sensible, since it restricts the search space, but users can prevent this behavior by disabling the metis_instantiate_undefined option, in which case the variables are replaced by ‘’.
Instantiation of Isabelle Facts.
The translation procedure yields a list of Isabelle facts and instantiations of their free variables. A single fact may possess multiple instantiations if it is used multiple times in the proof. To reduce the number of instantiations and to avoid duplicates, we merge instantiations whenever possible:
Definition 4
A (variable) instantiation for an Isabelle fact is a partial function from variable names to Isabelle terms in which all variable names occur in as free variables. Two instantiations for the same fact can be merged into if equals up to -equivalence for all shared variable names .
Merging can be used to avoid duplicates, since every instantiation can be merged with itself. Conversely, if two instantiations stem from the same clause and can be merged, they are equal up to -equivalence. This is because the translation procedure considers only variables occurring in , and all uninstantiated variables of are instantiated with or ‘’. As a result, the domains of the instantiations are exactly the variable names occurring in that correspond to a free variable in the fact.
By the above argument, if different instantiations can be merged, they must stem from different clauses. In practice, metis will still succeed with the merged instantiation, since the clausifier will split the instantiated fact into the instantiated clauses again. Consequently, merging does not lose information.
Example 6
Consider the following Isabelle fact :
[TABLE]
The clausifier splits it into two clauses. Suppose that both clauses are used in a metis proof and we inferred the instantiations and . Since the variable name is mapped to the same term in both instantiations, these can be merged to . By instantiating with , we obtain the following new fact:
[TABLE]
If metis is invoked again using this fact instead of the original fact, the clausifier will again split it into the two clauses, instantiated with and , respectively.
Since we use type inference instead of reconstructing type information, the translation procedure may produce Isabelle terms with overly generic types. Given that types can be displayed in Isabelle (e.g., using the show_types option) and overly generic types can prevent the folding of abbreviations, we attempt to concretize the types of the terms of a variable instantiation for a fact . We achieve this through type unification [28]: The types of the terms of are unified with the types of the corresponding variables in , and the unifier is then applied to all the types contained in the terms of . It is crucial to find a single unifier for the entire instantiation , since the same type variable may appear in multiple types of the free variables in , and they must be synchronized.
Finally, if instantiations have been found, metis suggests that the user replaces the current metis call with a new call using the instantiated facts. Each instantiated fact is displayed by its name together with an annotation that specifies to instantiate it with the terms of the corresponding instantiation. If there are multiple instantiations of a single fact, the fact is displayed multiple times. By replacing the call, the user may get a more readable and faster proof.
5 Sledgehammer Extension
Since metis calls are usually generated by Sledgehammer, we also extend Sledgehammer’s proof reconstruction module to enable direct generation of proofs with instantiated facts. By using Sledgehammer’s preplay module, this extension can not only generate faster and more readable proofs but also transform timeouts into successes as well as produce simpler proofs using other proof methods than metis.
Preplay with Instantiations.
Sledgehammer’s proof reconstruction is started after an external ATP has found and minimized a proof. The facts referenced in the ATP proof are then used to generate an Isabelle proof. The metis extension we presented in Section 4 can now be invoked to infer the instantiations of these facts. To do so, metis must first succeed—both the Metis proof (for inferring the instantiations) and the reconstruction in Isabelle (for ensuring that the instantiations are type-correct, if an unsound type encoding was used) are essential.
After instantiating the facts, Sledgehammer’s preplay tries out the proof with the instantiated facts. Even though metis has already found a proof, this is beneficial for a number of reasons:
The preplay module includes an additional minimization tool that repeatedly invokes a successful proof method with subsets of the facts. Its purpose is to reduce the number of facts and thereby simplify the generated Isabelle proof. Instantiated facts might be redundant and, consequently, can be eliminated by this minimization. Consider the goal and the fact . This fact is translated to two Metis clauses, corresponding to the two implications and . Since both clauses are equal up to the naming of variables, Metis uses only the first one to prove both directions of the goal. Accordingly, the instantiations and are inferred, resulting in two instantiated facts. Since only one of these is necessary, the minimization tool removes one of them. 2. 2.
Via a mechanism called try0, preplay tries a variety of proof methods [6], including multiple metis calls with different options (describing the encoding of types and -abstractions) and other standard proof methods, such as simp [26], blast [30], and auto [29]. This can result in even faster Isabelle proofs, since other proof methods are frequently faster than metis but sometimes require instantiated facts to find any proof at all (as demonstrated in Example 3). Additionally, Isabelle proofs resulting from methods such as simp, blast, and auto are often simpler and more readable, since they do not require the user to provide all the necessary facts for the proof. Minimization removes the facts that are irrelevant for these proof methods. 3. 3.
According to Theorem 4.1, a new Metis proof derived from the instantiated Metis clauses is possible. However, this does not necessarily extend to Isabelle proofs using metis. A new metis proof using the instantiated Isabelle facts may not be possible, since their encoding is not guaranteed to produce the same Metis clauses again. This is mostly due to -conversion (as demonstrated in Example 7). This rarely happens in practice but is nonetheless one reason to preplay the new metis proof.
During preplay, each proof method is executed for a predetermined duration (called the preplay time limit, which defaults to s). The instantiation of facts often speeds up proofs, possibly turning timeouts into successes. However, in order to instantiate the facts, metis must first succeed. Consequently, the metis call used to infer the instantiations is given more time than the proof methods during preplay. More precisely, it is executed for five times the preplay time limit (i.e., s by default).
An alternative would be to increase the preplay time limit in the first place. However, this would work less well in practice, because users typically insert the generated proofs directly into the Isabelle proof text. Given that these are re-executed each time the text is processed, multiple slow proofs could substantially increase the overall processing time.
Our Sledgehammer extension can be controlled through the three-valued Sledgehammer option instantiate. If this option is set to false, the extension is disabled. If the option is set to true, metis is started with the metis_instantiate option enabled directly after an ATP has found and minimized a proof, at the beginning of Sledgehammer’s proof reconstruction module. In most cases, this leads to Isabelle proofs with instantiated facts; otherwise, a proof without instantiations is displayed. The latter case may occur if metis is unable to find a proof (but, e.g., auto is able to), the proof does not use any facts, or there are no instantiations (e.g., if the facts contain no free variables).
Finally, if the option is set to smart, the instantiation of facts is started only if preplay failed (i.e., no proof method was successful within the preplay time limit). If the instantiation was successful, preplay is invoked once more with the instantiated facts, as usual. Therefore, Isabelle proofs with instantiated facts are displayed only if metis takes more time than the preplay time limit to prove the goal from the original facts and instantiation can speed up the proof so that it takes less time than the preplay time limit (as demonstrated in Example 2).
Although setting the option to true leads to faster, simpler, and more readable proofs, it is too disruptive. Most users, most of the time, are satisfied with uninstantiated facts, which are less verbose. They are pleased if they occasionally get a proof with instantiated facts that would have failed otherwise. Therefore, we make smart the default.
Extensionality for Metis.
Extensionality states that two functions are equal if they yield the same results for the same arguments. In Isabelle, extensionality is a basic axiom called ext:
[TABLE]
Since Metis targets first-order logic, it is unaware of this principle. Instead, ext must be passed to metis (which uses a first-order encoding) when extensionality is required. Since this enlarges the search space, this is not done by default.
However, the need for extensionality can change as facts are instantiated. In other words, ext may not be necessary for a proof derived from the original facts, but it may be necessary for a proof derived from the instantiated facts, and vice versa. This phenomenon is mostly due to Isabelle’s application of -conversion.
Example 7
Recall the fact , and suppose that we want to prove the following goal:
[TABLE]
To encode the -abstractions, we use -lifting [19] and thus introduce a new supercombinator with the definition , leading to a new goal:
[TABLE]
We invoke metis, which infers that should be instantiated with and should be instantiated with . As described in Section 4, is replaced by again, resulting in the following instantiated fact:
[TABLE]
Notice that Isabelle applied -reduction, resulting in the modified -abstraction . If we attempt a proof with metis and -lifting again, this -abstraction will be encoded as another supercombinator with the definition , where is the Skolem constant introduced for the existentially quantified variable . Using ’s and ’s definitions, Metis can prove that for every , but extensionality is needed to conclude that and thus to exchange the two terms and prove the goal. Therefore, metis can prove the goal from the instantiated fact only if we add the fact ext.
Thus, ext may be necessary for a proof derived from the instantiated facts. Conversely, if -conversion eliminates -abstractions, ext might be unnecessary for a proof derived from the instantiated facts even though it is needed for a proof derived from the original facts. Therefore, we also extend the preplay module so that a few metis calls with the additional fact ext are tried after other proof methods have failed, including multiple metis calls with different options. These new metis calls use -lifting as the encoding of -abstractions, since -lifting benefits more from ext than the combinators [22].
The metis calls with the fact ext are also useful regardless of instantiations, since some ATPs, such as Zipperposition [38], are based on higher-order logic and already include extensionality. As a result, their proofs do not need ext, and proof reconstruction might be unsuccessful without the new metis calls.
6 Evaluation
Our empirical evaluation is based on the repository revision d3c0734059ee (October 25, 2024) of Isabelle and 4082096ade5a (October 25, 2024) of the Archive of Formal Proofs [9].
Setup.
We performed the evaluation using the Slurm batch system [21] on the computer resources of the Institute for Informatics at Ludwig-Maximilians-Universität München. We requested GiB of RAM and 8 logical processors (CPU threads). Given that not all Sledgehammer steps and ATPs are deterministic and time limits are used, the results are not entirely reproducible. To increase reproducibility, we used a fresh Isabelle installation and reset the state of the learning-based relevance filter MaSh [8] before each evaluation run.
We used the testing and evaluation tool Mirabelle [16], which is included in Isabelle. Mirabelle applies a selected action to each selected goal. We used the same Archive entries as Desharnais et al. [16], from which we randomly selected goals per entry, resulting in a total of goals. As the action, we invoked Sledgehammer with several options:
- –
The try0 option was disabled so that Sledgehammer’s preplay exclusively tried metis. This was done to test metis extensively, since we extended this proof method, and to keep the evaluation feasible in reasonable time, since other proof methods frequently caused the Isabelle process to abort for technical reasons.
- –
The provers option for specifying the external ATPs was set to use the high-performance [16] superposition provers E, Vampire, and Zipperposition. We excluded satisfiability-modulo-theories (SMT) solvers since these support theories, such as linear arithmetic, that are not built into metis.
- –
The strict option was enabled to force the use of sound type encodings for the external ATPs to produce type-correct proofs that should be reconstructable.
We performed two evaluation runs, with Sledgehammer’s instantiate option set to smart in the first run and to true in the second run. From the Mirabelle output of each run, we extracted the number of successful Isabelle proofs with and without instantiations, the average execution times of Sledgehammer and metis, and the kinds of metis calls used in the generated Isabelle proofs. With this information, we try to answer the following research questions:
Is it possible to reconstruct additional proofs by instantiating the facts, and in how many cases is a proof derived from the instantiated facts impossible? 2. 2.
What are the advantages of enabling Sledgehammer’s instantiate option? That is, how often are Isabelle proofs with instantiated facts actually presented to users, how much longer do Sledgehammer executions take, and how much faster are Isabelle proofs? 3. 3.
What do Sledgehammer-generated metis calls look like after instantiating the facts, and what are the benefits of adding the fact ext?
Results.
We consider only Sledgehammer invocations where an external ATP has found a proof. Sledgehammer’s proof reconstruction module initially attempts to generate a one-line proof consisting of a single proof method invoked with facts, which can now have instantiations thanks to our extensions. If this attempt is unsuccessful, Sledgehammer tries to transform the ATP proof into an Isar proof [6]. If this also fails, the proof reconstruction is considered to have failed. The number of occurrences of each of these outcomes in each evaluation run is shown in Table 1.
In the first evaluation run, there were a total of Sledgehammer invocations yielding ATP proofs. Whereas ATP proofs () were reconstructed directly as one-line proofs without instantiations, additional ATP proofs () could be reconstructed by instantiating the facts and thereby speeding up the metis proofs. This number may seem low, but this corresponds to of cases where there was no one-line proof before. The remaining of ATP proofs could not be reconstructed as one-line proofs. This number may seem alarmingly high, but detailed Isar proofs were available in of the cases, and in practice the try0 option is usually enabled, which leads to further proof methods being tried in addition to metis. We note that our approach brings similar benefits to Sledgehammer’s success rate as Isar proof reconstruction.
In the second evaluation run, there were a total of Sledgehammer invocations with ATP proofs. In cases (), Sledgehammer suggested a one-line proof with instantiated facts. In cases (), Sledgehammer suggested a one-line proof without instantiated facts. The raw evaluation data reveals that in most of these cases, the proof did not use any facts or there were no instantiations.
However, sometimes the instantiations were successfully inferred, but preplay failed to generate a metis proof derived from the instantiated facts. According to the raw data, there were such cases () in the first evaluation run and () in the second. There are several reasons for this, including the following:
- –
The instantiations did not speed up the metis proof enough to take less time than the preplay time limit. The instantiations could have even prolonged the proof, since they modify the problem and could affect Metis’s heuristics.
- –
Isabelle’s extremely flexible syntax allows ambiguities and does not guarantee that all terms can be parsed back [6].
Table 2 shows the average execution times of Sledgehammer invocations that successfully generated a one-line metis proof. The average execution time of Sledgehammer increased by ms () from the first to the second evaluation run, which can be explained by the additional time required for instantiating facts. However, this did not have a substantial impact on the overall duration of an average Sledgehammer execution. By contrast, the average execution time of metis decreased by ms (), indicating that the instantiation of facts markedly speeds up proofs. Notice that in about of cases ( and in Table 1), the metis calls generated in the two evaluation runs were both without or both with instantiations. Consequently, for the remaining of cases, it can be deduced that the average execution time decreases by more than . If those proofs are inserted into an Isabelle proof text, this improvement applies every time the text is processed. In particular, every one of the frequent regression tests of the Archive of Formal Proofs with its more than metis calls could benefit.
In Sledgehammer’s preplay, a series of metis calls are tried, each involving different options that describe the encoding of types and -abstractions. Initially, the default options (type encoding [7] and combinators [37]) are tested, resulting in a standard metis call. If this fails, Sledgehammer tries multiple alternative options, selecting the fastest metis call with options. If this also fails, Sledgehammer tries the new metis calls with the additional fact ext. The number of occurrences of each of these kinds of metis calls in one-line proofs in each evaluation run is shown in Table 3.
The number of standard metis calls increased from the first evaluation run () to the second (). This observation suggests that Metis’s proof search becomes more efficient after the instantiation of facts, since special encodings are needed less often. Moreover, adding ext was effective, since it was used in and of the proofs, respectively, which would have failed otherwise.
Regarding our research question 1, the results show that Sledgehammer can reconstruct additional one-line proofs by instantiating the facts, and that a proof derived from the instantiated facts is almost always possible. Regarding question 2, we observe that enabling Sledgehammer’s instantiate option leads to proofs with instantiations in of the cases, which may help users understand the generated proofs. Sledgehammer executions also take a bit longer, but the execution time of the generated Isabelle proofs is substantially reduced, which is much more important. Regarding question 3, we see that there are more standard metis calls without options after instantiating the facts and that adding the fact ext brings similar benefits to Sledgehammer’s success rate as instantiating facts.
7 Related Work
The Metis ATP [20], on which our work is based, was developed by Hurd with the goal of integrating it into the HOL proof assistant [18]. Metis was integrated in Isabelle by Paulson and Susanto [32]. Metis’s fine-grained proofs were particularly useful to us. Beyond the HOL and Isabelle integrations, there is also a Metis proof checker in Agda developed by Prieto-Cubides and Sicard-Ramírez [33].
Our tool joins the ranks of various proof analysis tools. The GAPT (General Architecture for Proof Theory) framework by Ebner et al. [17] consists of data structures, algorithms, parsers, and more, with the aim of supporting proof theory applications and ATPs. Other proof-manipulating frameworks are ProofCert by Miller and colleagues [24, 14] and Dedukti by Dowek and colleagues [1, 11].
Our work is also loosely related to the experimental reconstruction of ATP proofs as detailed Isabelle proofs, or Isar proofs [40], by Blanchette et al. [6].
8 Conclusion
We extended Isabelle’s metis proof method and the Sledgehammer tool to infer variable instantiations from proofs and present them to users. This speeds up Sledgehammer’s proof reconstruction and increases its success rate. It also helps users to understand the proof without inspecting all of its details.
We see three main directions for future work. First, we could provide type annotations when parsing of the instantiations is ambiguous. Second, we could provide instantiations not only for one-line proofs but also for individual steps in detailed Isar proofs. Third, we could extend the approach to more ATPs in order to obtain variable instantiations even when metis without instantiations fails. The superposition provers E, SPASS, Vampire, and Zipperposition are difficult to integrate, because they fail to preserve the connection between the variables before and after clausification. The metis proof method’s clausifier, by contrast, preserves variable names. As for SMT solvers, some of them, including cvc5, can be asked to output the instantiations they needed to show unsatisfiability. Then there is no need to analyze the proofs to infer the instantiations.
8.0.1 Acknowledgments.
We thank Xavier Généreux, Elisabeth Lempa, Jannis Limperg, Kirstin Peters, Mark Summerfield, and the anonymous reviewers for suggesting various textual improvements. Blanchette’s research was cofunded by the European Union (ERC, Nekoka, 101083038). Views and opinions expressed are however those of the authors only and do not necessarily reflect those of the European Union or the European Research Council. Neither the European Union nor the granting authority can be held responsible for them.
8.0.2 Disclosure of Interests.
The authors have no competing interests to declare that are relevant to the content of this paper.
The reference list from the paper itself. Each links out to its DOI / PubMed record.
- 1[1] Assaf, A., Burel, G., Cauderlier, R., Delahaye, D., Dowek, G., Dubois, C., Gilbert, F., Halmagrand, P., Hermant, O., Saillard, R.: Dedukti: a logical framework based on the λ \lambda Π \Pi -calculus modulo theory. Co RR abs/2311.07185 (2023). https://doi.org/10.48550/ARXIV.2311.07185
- 2[2] Bachmair, L., Ganzinger, H.: On restrictions of ordered paramodulation with simplification. In: Stickel, M.E. (ed.) CADE-10. LNCS, vol. 449, pp. 427–441. Springer (1990). https://doi.org/10.1007/3-540-52885-7_105
- 3[3] Barbosa, H., Barrett, C.W., Brain, M., Kremer, G., Lachnitt, H., Mann, M., Mohamed, A., Mohamed, M., Niemetz, A., Nötzli, A., Ozdemir, A., Preiner, M., Reynolds, A., Sheng, Y., Tinelli, C., Zohar, Y.: cvc 5: A versatile and industrial-strength SMT solver. In: Fisman, D., Rosu, G. (eds.) TACAS 2022, Part I. LNCS, vol. 13243, pp. 415–442. Springer (2022). https://doi.org/10.1007/978-3-030-99524-9_24
- 4[4] Bhayat, A., Suda, M.: A higher-order Vampire (short paper). In: Benzmüller, C., Heule, M.J.H., Schmidt, R.A. (eds.) IJCAR 2024, Part I. LNCS, vol. 14739, pp. 75–85. Springer (2024). https://doi.org/10.1007/978-3-031-63498-7_5
- 5[5] Blanchette, J.: Hammering away: A user’s guide to Sledgehammer for Isabelle/HOL (2025), https://isabelle.in.tum.de/website-Isabelle 2025/dist/Isabelle 2025/doc/sledgehammer.pdf
- 6[6] Blanchette, J.C., Böhme, S., Fleury, M., Smolka, S.J., Steckermeier, A.: Semi-intelligible Isar proofs from machine-generated proofs. J. Autom. Reason. 56 (2), 155–200 (2016). https://doi.org/10.1007/S 10817-015-9335-3
- 7[7] Blanchette, J.C., Böhme, S., Popescu, A., Smallbone, N.: Encoding monomorphic and polymorphic types. Log. Meth. Comput. Sci. 12 (4) (2016). https://doi.org/10.2168/LMCS-12(4:13)2016
- 8[8] Blanchette, J.C., Greenaway, D., Kaliszyk, C., Kühlwein, D., Urban, J.: A learning-based fact selector for Isabelle/HOL. J. Autom. Reason. 57 (3), 219–244 (2016). https://doi.org/10.1007/s 10817-016-9362-8