Towards Ranking Geometric Automated Theorem Provers

TL;DR

This paper discusses the development of a ranking system for geometric automated theorem provers, emphasizing the need for standardized testing, common problem descriptions, and quality measures to evaluate and compare their performance.

Contribution

It proposes a framework for ranking geometric theorem provers through a common test bench, standardized problem descriptions, and quality metrics.

Findings

Highlights the importance of evaluation criteria for geometric provers

Identifies the need for a common test bench and problem repository

Suggests potential improvements in prover performance and reliability

Abstract

The field of geometric automated theorem provers has a long and rich history, from the early AI approaches of the 1960s, synthetic provers, to today algebraic and synthetic provers. The geometry automated deduction area differs from other areas by the strong connection between the axiomatic theories and its standard models. In many cases the geometric constructions are used to establish the theorems' statements, geometric constructions are, in some provers, used to conduct the proof, used as counter-examples to close some branches of the automatic proof. Synthetic geometry proofs are done using geometric properties, proofs that can have a visual counterpart in the supporting geometric construction. With the growing use of geometry automatic deduction tools as applications in other areas, e.g. in education, the need to evaluate them, using different criteria, is felt. Establishing a ranking among geometric automated theorem provers will be useful for the improvement of the current methods/implementations. Improvements could concern wider scope, better efficiency, proof readability and proof reliability. To achieve the goal of being able to compare geometric automated theorem provers a common test bench is needed: a common language to describe the geometric problems; a comprehensive repository of geometric problems and a set of quality measures.

Tables

Table 1. Table 1: Comparing GATPs

	Coq	GCLC			GeoGebra	OGP
	AM	AM	WM	GBM	BG	WM
GEO0080	0.74	0.012	0.016	time-out	0.615	0.028
GEO0226	—	0.748	1.62	time-out	time-out	0.075
GEO0228	—	0.008	0.012	0.036	0.207	0.015
GEO0237	time-out	time-out	0.048	time-out	0.264	0.105
GEO0238	time-out	0.032	0.024	0.092	0.18	0.102
GEO0240	—	time-out	0.056	time-out	1.314	0.030

Full text

The geometry automated deduction area differs from other areas by the strong connection between the axiomatic theories and its standard models. In many cases the geometric constructions are used to establish the theorems’ statements, geometric constructions are, in some provers, used to conduct the proof, used as counter-examples to close some branches of the automatic proof. Synthetic geometry proofs are done using geometric properties, proofs that can have a visual counterpart in the supporting geometric construction.

With the growing use of geometry automatic deduction tools as applications in other areas, e.g. in education, the need to evaluate them, using different criteria, is felt. Establishing a ranking among geometric automated theorem provers will be useful for the improvement of the current methods/implementations. Improvements could concern wider scope, better efficiency, proof readability and proof reliability.

To achieve the goal of being able to compare geometric automated theorem provers a common test bench is needed: a common language to describe the geometric problems; a comprehensive repository of geometric problems and a set of quality measures.

1 Introduction

The first geometric automated theorem provers proposed, that came as early as in 1959 [14], adapt general-purpose reasoning approaches developed in the field of artificial intelligence, automating the traditional geometric proving processes. In order to avoid combinatorial explosion while applying postulates, many suitable heuristics, e.g. adding auxiliary elements to the geometric configuration, have been developed. Although being able to produce readable proofs, the proposed methods were very narrow-scoped and not efficient [35].

Recent results using this approach, like the ArgoCLP theorem prover, based on coherent logic [31], or the deductive database approach [12] having been proposed with different rates of success in different classes of geometric problems.

The algebraic methods, such as the characteristic set method [6, 35], the polynomial elimination method [33], the Gröbner basis method [19], and the Clifford algebra approach [21], reduce the complexity of logical inferences by computing relations between coordinates of geometric entities. What is gained in efficiency and wider scope is lost in the connection of the algebraic proof and the geometric reasoning. These methods are broad-scope, efficient, but if, eventually, a proof record is produced, it will be a very complex algebraic proof [8, 35]. This is still an active area of research [3, 15, 16, 20, 38].

In order to combine the readability of synthetic methods and efficiency of algebraic methods, some approaches, such as the area method [10, 15], the full angle method [11] represent geometric knowledge in a form of expressions with respect to geometric invariants. These methods are broad-scoped (less than the algebraic), efficient (less than the algebraic) and capable of producing readable proofs. An implementation as a Coq111https://coq.inria.fr/ contribution is such that it can have theirs proofs verified [4, 15].

When considering ranking the Geometric Automated Theorem Provers (GATP) we have to consider some, somehow opposing, goals: scope, the geometries being considered and what kind of problems are provable; efficiency, the time needed to complete the proof; readability of the proof produced (if any); and, with the help of proof assistants, the reliability of generated proofs [18, 25].

To be able to compare the different methods and implementations, a test bench must be defined, a common language to state the geometric theorems, a comprehensive repository of geometric problems and a set of measures of quality capable of assessing the GATPs in different classes, such as: scope; efficiency; readability; reliability of generated proofs.

Overview of the paper. The paper is organised as follows: first, in Section 2, the test bench is discussed. In Section 3, the different measures of quality are discussed. Final conclusions are drawn and future work is foreseen in Section 4.

2 Test Bench

In order to implement a test bench, the Thousands of Geometric problems for geometric Theorem Provers222http://hilbert.mat.uc.pt/TGTP/ (TGTP) platform presents itself as a solid foundation to fulfil such purpose. TGTP aims to provide the automated reasoning in geometry community with a comprehensive and easily accessible, library of GATP test problems [24], it already provides a centralised common repository of geometric problems333As of 2018–04–21 there were 236 problems., an unambiguous reference mechanism, textual and geometric search mechanism and the problems are kept in the i2gatp common format [26]. Moreover, TGTP provides, as part of its infrastructure, implementations of several methods, namely: GCLC444http://poincare.matf.bg.ac.rs/~janicic/gclc/ implementations of Wu’s method, Gröbner basis method and the area method; and a Coq implementation of the area method555https://github.com/coq-contribs/area-method [22]. Statistical and performance information is also supplied for all implementations, as well as a proof status for each geometric problem.

With a working test bench in operation, an interesting goal to pursue would be to operationalize a competition between GATPs, similar to CASC [32]. Its ultimate goal, like in CASC, would be to encourage researchers to improve existing GATPs and implement new ones.

For example, in table 1 the CPU times666Linux 4.9.0-2-amd64 #1 SMP Debian 4.9.18-1 (2017-03-30), Intel(R) Core(TM) i7-4770 CPU @ 3.40GHz. The used “time-out” threshold was 60s, but it can be configured for any given value. needed to complete the proofs for some TGTP problems are presented. The GATPs used were: Coq;777For some of the examples it was not possible to have the transcription of the problem in Coq GCLC; GeoGebra; OpenGeoProver (OGP) with the methods: area method (AM); Wu’s method (WM); Gröbner Basis’ method (GBM); BotanaGiac (BG) [2]. The OGP and GCLC implementations of the Wu’s method stand-off as the only implementations that were able to prove all the conjectures, with GCLC having a marginal advantages over OGP.

But, apart CPU times, when ranking GATPs other issued must be considered: the scope, the proof readability and proof reliability.

The GCLC implementation of the area method is the only GATP providing a readable synthetic geometric proof.

The GCLC implementation of the Wu’s method is the one with the wider scope (for this small set of examples), being able to prove all the problems.

The Coq implementation of the area method is the only GATP formally verified [4, 15]. The Coq proof assistant relies on a small kernel, extensively verified, against whom all the other layers are formally checked. The Coq implementation of the area method is thus formally verified. For all the other GATPs the implementations trustworthiness relies on the GATPs code as a whole, without any formal verification of its correctness. This issue is more important when the algebraic implementations are to be considered, in those cases its black box nature, with only a yes/no answer, prevent the user to manually check the proof produced. Any new implementation should be extensively checked before it can be considered finished.

The intended use is also important. For example, in education two uses of a GATP are foreseen, the validation of a given conjecture and/or construction [2, 17] and the proof itself as object of study.

For the first case (validation), the proof is not needed, only a formal validation is needed, speed is the most important measure of quality. Rowe [29] and, later, Stahl [30] studied this issue, defining the “Post-Teacher Question Wait-Time”, i.e. the time a high-school class will wait in silence after a teacher place a question. This is a (surprisingly!?) short span of time, a good response time is less then 1.5s, a fair response time will be less then 3s, everything above that would be unsuitable for a high-school classroom. Looking in table 1, we can see that the implementations of the algebraic methods (with the exception of GCLC implementation of the Gröbner basis) gave good results for all but two cases. One result in the border between good and fair and a “time-out”.

For the second case a readable proof is needed, i.e. a synthetic or semi-synthetic geometric proof readable, by teachers (at least) and students. For the first case the algebraic methods seems to be the best one, for the second case they are completely useless, the proofs produced are algebraic, unrelated with the geometric construction in consideration. The synthetic or even semi-synthetic should be considered instead.

As far as the authors of this paper know, there are two proposals to measure the readability of a proof: Chou et al. [9] proposed a way to measure how difficult a formal proof is (using the area method) based on the time and the number of steps needed to perform it; de Bruijn proposed the de Bruijn factor [5, 34],888https://www.cs.ru.nl/~freek/factor/ the quotient of the size of corresponding informal proof and the size of the formal proof. Apart this quantitative measures some qualitative considerations and also considerations about the audience in consideration led to a schema, proposed in [27], to classify the proofs produced by GATPs:

no readable proof; 2. 2.

non-synthetic proof (i.e. a proof without a correspondent geometric description, e.g. algebraic methods); 3. 3.

semi-synthetic proof with a corresponding prover’s language rendering; 4. 4.

(semi-)synthetic proof with a corresponding natural language rendering; 5. 5.

(semi-)synthetic proof with a corresponding natural language and visual renderings [36, 37].

The synthetic proofs with a corresponding natural language and visual renderings is the most difficult to attain, but also the more desirable, e.g. in an educational setting.

3 Ranking GATP

To be able to compare the different methods and implementations, alongside a standard test bench (see Section 2), we must define a set of measures of quality capable of assessing the GATPs in different classes: scope; efficiency; readability; reliability of generated proofs.

Scope

To measure the scope of a GATP, one should consider:

• •

which geometries are allowed by the GATP—despite the fact that most GATPs deal only with Euclidean geometry, some exist that prove geometric problems in non-Euclidean geometries [7, 39];

• •

what kind of problems are provable—as an example, the area method uses geometric invariants as basic quantities to prove theorems, each of which is used to deal with different geometric relations, hence allowing certain theorems to be easily proved [9].

Considering the existing methods, algebraic ones have the broadest scope. Not only have they been used to prove theorems in Euclidean and non-Euclidean geometry, but, for every geometry, the range of difficulty of the problems proved is very wide [7]. The earlier synthetic approaches were narrow scoped, e.g. the GEOM [13] only dealt with a limited set of geometric elements and relations [9]. In between lie semi-synthetic methods and coherent logic based methods. Semi-synthetic methods, starting with the area method which is complete for constructive geometry [10, 11, 15], with its geometric invariant, the signed area, allow many problems with relations like incidence and parallelism to be proved. Adding another geometric invariant, the Pythagoras difference, allows the problems with relations like perpendicularity and congruence of line segments to be easily proved. Adding other geometric invariants such as the full-angle (which gives its name to the full-angle method), the volume and the vector, allows the demonstration of an ever increasing range of theorems [9].

Synthetic and semi-synthetic methods scope may be influenced by the use of deductive databases [12, 13]. Indeed, as stated in [12], unexpected results may be obtained, some of which are possibly new.

Efficiency

The purpose of a GATP is, in addition to prove geometric conjectures, that these are obtained efficiently. By and large, although other resources may be involved, efficiency is related to time and memory space: we look for algorithms/implementations capable of fulfilling a proof in a reasonable amount of time and space.

Time is indeed the natural way to measure efficiency since it is used extensively, if not exclusively, throughout the literature [7, 8, 10, 11, 12, 15]. and, for obvious reasons, in a competition such as CADE ATP System Competition (CASC). Moreover, such measure is of paramount importance when considering an educational environment.

Note however that when authors state something about GATP times, they do so in their settings, i.e. computer and operating system used, somehow restricting the usefulness of these results. The existence of a free and open platform where different GATPs can be tested on equal terms proves to be of utmost importance. In table 1 the examples were tested under the same system being comparable, an ongoing work will allow using all the conjectures in TGTP to test all those GATPs under the same system.

Regarding space, the authors are unaware of any study, presumably because these physical constraints are nowadays less important. Besides, from the users point of view time is the most important factor.

Readability

Until recently, proofs in mathematics were solely made and verified by humans. With the advent of computers and automated reasoning that is no longer the case. As in some settings readability of a proof (by a human) is of utmost importance, such characteristic is considered crucial. Indeed, in an educational setting the proof is an object of learning by itself and the ability of a GATP to produce a synthetic proof, with the usual geometric inference rules, is of fundamental importance to its usability.

TGTP both the quantitative and qualitative measures should be introduce in such a way that a user can filter the information in the way that best fit his/her needs.

Reliability

By reliability is meant the confidence that we have in the proofs made by a given prover. Is the prover correctly implemented?

Using proof assistants like Coq, or Isabelle,999https://isabelle.in.tum.de/ the implementation of a given prover can be formally established, e.g. in [22] a implementation of the area method within Coq is described, where all the properties of the geometric quantities required by the area method are verified, demonstrating the correctness of the system, reducing concerns of reliability, to the trustworthiness of respective proof assistants [15, 22].

For the other implementations, the trustworthiness of GATP relies on the implementation code as a whole and in the extensive testing of the new implementations, e.g. using a workbench like TGTP.

4 Conclusion and Future Work

When regarding the area of geometric automated deduction the evaluation of automated provers is no longer a mono-objective problem, time considerations are important but scope and readability of the proofs are also very important.

Maybe the question “what is the best GATP of them all”, can not be answered, but at least we should have some partials answers when looking for a GATP that fit a particular goal.

The intended audience is also a point to be considered, e.g the narrow scope provers with specific heuristics, closing the circle with the early AI approaches of the 1960s, may be not relevant to an expert in the field of automated theorem proving, but the definition of sub-languages for educational use, could led to narrow scope GATPs with good qualities when readability of the proofs is to be considered, in specific educational settings.

We need a common workbench: a common language to describe the geometric problems; a large set of problems and a set of measures. Building on TGTP test bench [24], i2gatp common language [26] and CASC, CADE ATP System Competition [32], the first two need to be improved and better integrated, and adapting the ideas of the last, a common test bench should be built, with the goal of helping GATPs developers to improve their systems, to better fit the needs of the different communities of users.