More Effective Ontology Authoring with Test-Driven Development

TL;DR

This paper introduces a logic-based, test-driven approach to ontology authoring that improves efficiency and reduces errors, demonstrated through a new Protégé plugin called TDDonto2.

Contribution

It presents novel TDD algorithms covering all OWL 2 class expressions, with a formal foundation and implementation in TDDonto2, enhancing ontology authoring effectiveness.

Findings

TDDonto2 is faster than traditional interfaces for medium and large ontologies.

Users make fewer errors and complete tasks more efficiently with TDDonto2.

The approach is validated through efficiency and user studies.

Abstract

Ontology authoring is a complex process, where commonly the automated reasoner is invoked for verification of newly introduced changes, therewith amounting to a time-consuming test-last approach. Test-Driven Development (TDD) for ontology authoring is a recent {\em test-first} approach that aims to reduce authoring time and increase authoring efficiency. Current TDD testing falls short on coverage of OWL features and possible test outcomes, the rigorous foundation thereof, and evaluations to ascertain its effectiveness. We aim to address these issues in one instantiation of TDD for ontology authoring. We first propose a succinct, logic-based model of TDD testing and present novel TDD algorithms so as to cover also any OWL 2 class expression for the TBox and for the principal ABox assertions, and prove their correctness. The algorithms use methods from the OWL API directly such that reclassification is not necessary for test execution, therewith reducing ontology authoring time. The algorithms were implemented in TDDonto2, a Prot\'eg\'e plugin. TDDonto2 was evaluated on editing efficiency and by users. The editing efficiency study demonstrated that it is faster than a typical ontology authoring interface, especially for medium size and large ontologies. The user evaluation demonstrated that modellers make significantly less errors with TDDonto2 compared to the standard Prot\'eg\'e interface and complete their tasks better using less time. Thus, the results indicate that Test-Driven Development is a promising approach in an ontology development methodology.

Tables

Table 1. Table 1 : Classification time in seconds (“Classif. (s)”), average number of characters for names ( a 𝑎 a ) of class ( C ) and object properties ( OP ) and hierarchy depth ( b 𝑏 b ), and instances ( c 𝑐 c ) used in the editing efficiency computation. M1-M3: mock ontologies where values are set to assess their effects on editing efficiency.

	AWO	Pizza	DMOP	M1	M2	M3
Classif. (s)	0.81	0.1	1196.53	100	500	25
$a_C$	7.06	13.07	21.09	15	15	23
$b_C$	2	4.86	8.39	6	12	6
$a_{OP}$	9.4	11.63	14.14	12	12	15
$b_{OP}$	1.2	1.5	2.2	2	3	1.5
$c$	0	6.4	19.03	10	10	19

Table 2. Table 2 : Statistics corresponding to claims C1, C2, C3.

Task	Group A	Group B	Total
	(basic editor first)	(TDDonto2 first)
Basic editor
TBox1	13 (100%)	10 (83%)
ABox1	13 (100%)	9 (75%)	90%
TDDonto2
TBox2	13 (100%)	12 (100%)
ABox2	12 (92%)	10 (83%)	94%
Total	98%	85%

Table 3. (a) Degree of completing the tasks per each group.

Task	Group A	Group B	Total
	(basic editor first)	(TDDonto2 first)
Basic editor
TBox1	13 (100%)	10 (83%)
ABox1	13 (100%)	9 (75%)	90%
TDDonto2
TBox2	13 (100%)	12 (100%)
ABox2	12 (92%)	10 (83%)	94%
Total	98%	85%

Table 4. (b) Correctness of completing the tasks per each group.

Task	Group A	Group B	Total
	(basic editor first)	(TDDonto2 first)
Basic editor
TBox1	61/117 (52%)	63/90 (70%)
ABox1	26/39 (67%)	20/27 (74%)	62%
TDDonto2
TBox2	98/117 (84%)	96/108 (89%)
ABox2	31/36 (86%)	30/30 (100%)	88%
Total	70%	82%

Table 5. (c) Average time (in minutes) of completing the tasks per each group.

Task	Group A	Group B	Total
	(basic editor first)	(TDDonto2 first)
Basic editor
TBox1	19.00 min	19.19 min
ABox1	4.92 min	4.67 min	12.27 min
TDDonto2
TBox2	8.15 min	7 min
ABox2	2.86 min	2.60 min	5.38 min
Total	8.97 min	8.56 min

Equations

$$\text{mbox}{\text{em}pre-test}_O(A)=\{\begin{array}{ll}O⊬\top ⊑\perp & \text{ontology is consistent}\\ C\in V_C\text{ s.t. }C⊑\neg \perp & \text{ontology is coherent}\\ \Sigma (A)\subseteq \Sigma (O)& \text{axiom vocabulary}\\ & \text{elements present in }O\end{array}$$

$${\mathrm{test}}_O(A)=\{\begin{array}{ll}\mathrm{entailed}& \text{if }O⊢A\\ \mathrm{inconsistent}& \text{if }O\cup A⊢\perp \\ \mathrm{incoherent}& \text{if }\begin{array}{rl}& O\cup A⊬\perp \wedge (\exists C\in {\Sigma }_C(O))\wedge \\ & O\cup A⊢C⊑\perp \end{array}\\ \mathrm{absent}& \text{otherwise}\end{array}$$

$$\text{textsc}getInstances(C\sqcap \neg D)=\varnothing \wedge$$

$$\neg \text{textsc}isSatisfiable(C\sqcap \neg D)$$

$$O\cup (C⊑D)⊢C\sqcap \neg D⊑\perp ⟹O\cup (C⊑D)⊢N⊑\perp$$

$$⟹{\mathrm{test}}_O(C⊑D)=\mathrm{incoherent}$$

$$(\exists N\in {\Sigma }_C(O))N⊑C\wedge N⊑\neg D$$

$$⟺(\exists N\in {\Sigma }_C(O))N⊑C\sqcap \neg D$$

$$⟺(\exists N\in {\Sigma }_C(O))N\in \text{textsc}getSubClasses(C\sqcap \neg D)$$

$$⟺\text{textsc}getSubClasses(C\sqcap \neg D)\ne \varnothing$$

Full text

††journal: …

1 Introduction

Ontology engineering is facilitated by methods and methodologies, and tooling support for them. The methodologies are mostly information system-like, high-level directions, such as variants on waterfall and lifecycle development [1, 2], although more recently, notions of Agile development are being ported to the ontology development setting, e.g., [3, 4], including testing in some form [5, 6, 7, 8]. Now that most automated reasoners for OWL have become stable and reliable over the years, new methods have been devised that use the reasoner creatively in the support of the ontology authoring process. Notably, the OWL reasoner can also be used for examining negations [9, 5], checking the changes in entailments after an ontology edit [10, 11], and proposing compatible object properties for any two classes [12].

Such tools are motivated at least in part by the time-consuming trial-and-error authoring process, i.e., where a modeller checks consistency after each edit [13]. However, aforementioned methods and tools still require classification for each assessment step, which is unsustainable for large or complex ontologies due to prohibitively long classification times. Effectively, these modellers take a test-last approach to ontology authoring. In this respect, ontology engineering methodologies still lag behind software engineering methodologies both with respect to maturity and adoption [14].

There are a few recent attempts at explicitly incorporating automated testing with a test-first approach [15, 8], which is common in software engineering and known under the banner of test-driven development (TDD) [16]. To the best of our knowledge, there are three tools for TDD unit testing of ontologies [15, 8, 17] such that one can check whether an axiom is entailed before adding it. They exhibit two shortcomings that prevent potential for wider uptake: 1) certain axioms expressible in OWL 2 DL [18] are not supported as TDD tests, such as $\forall R.C\sqsubseteq D$, and 2) the outcome of a test can be only “pass” or “fail” with no further information about the nature of failure. Further, there has been no rigorous theoretical analysis of the techniques used for such ontology testing that avails of the automated reasoner pre-emptively. Yet, for modellers to be able to rely on reasoner-driven TDD for ontology authoring—as they do with test-last ontology authoring—such a theoretical foundation would be needed.

In this paper, we aim to fill this gap in rigour and coverage. We first propose a succinct logic-based model of TDD unit testing as a prerequisite. Subsequently, we generalise the piecemeal algorithms of [15] to cover also any OWL 2 class expression in the axiom under test for not only the TBox, as in [15], but also for the principal ABox assertions, and prove their correctness. These algorithms do not require reclassification of an ontology in any test after a first single classification before executing one or more TDD tests, and are such that the algorithms are compliant with any OWL 2 compliant reasoner. This is feasible through bypassing the ontology editor functionality (‘pressing the reasoner button’) and availing directly of a set of methods available from the OWL reasoners in a carefully orchestrated way. We have implemented the algorithms by extending one of the three TDD tools for ontologies, TDDOnto [15], into TDDonto2—also a Protégé 5 plugin—as a proof-of-concept to ascertain their correct functioning practically [19]111This open source plugin is accessible at https://github.com/kierendavies/tddonto2. This implementation was subsequently used in two evaluations. First, we devised a human-independent editing efficiency approach examining clicks, keystrokes, and reasoner invocation to compare the test-last with the basic Protégé 5 interface to the test-first with TDDonto2. TDDonto2 has a higher editing efficiency, i.e., takes less time, than the basic interface with a small difference for very small ontologies and substantially for medium to large ontologies. Second, we conducted a typical user evaluation to compare the basic Protégé interface to TDDonto2, which demonstrated that the modellers completed a larger part of the tasks with fewer mistakes in less time. Thus, TDD with the test-first approach and TDDonto2 is more effective than the common (test-last with Protégé) authoring approach.

In the remainder of the paper, we first provide motivations for why testing is applicable to ontologies (Section 2), and describe the main requirements. Section 3 describes related work. The first part of the main contributions are presented in Section 4, which is the model for testing and the main novel algorithms. The evaluations of its implementation are presented in Section 5. Section 6 discusses the work and we conclude in Section 7.

2 Motivations and requirements for a test-first approach

Test-Driven Development (TDD) in software engineering [16] is a methodology based on two rules:

1. write new code only if an automated test has failed, and
2. eliminate duplication. This induces a “red–green–refactor” pattern of development: first write a new test which fails, then write code which makes it pass with minimal effort, then remove resultant duplication and restructure if necessary. Tests thus serve to define desired functionality. The process is usually facilitated with a test harness that runs tests automatically and generates reports. TDD has been shown to improve code quality [20], especially in complex projects, and it is also shown to improve productivity [21]. In light of this, TDD may also be used for ontology development. In the next two subsections we first motivate where in the ontology authoring process tests could be useful and subsequently look at the broader picture of the ontology development processes.

2.1 TDD tests for ontologies

Ontologies, like computer programs, can become complex so that it is difficult for a human author to predict the consequences of changes. Automated tests are therefore useful to detect unintended consequences. As an illustrative example, suppose an author creates the following classes and subsumptions: $\mathtt{Giraffe}\sqsubseteq\mathtt{Herbivore}\sqsubseteq\mathtt{Mammal}\sqsubseteq\mathtt{Animal}$. The author then realises that not all herbivores are mammals, so shortens the hierarchy to $\mathtt{Herbivore\sqsubseteq Animal}$, thereby losing the $\mathtt{Giraffe}\sqsubseteq\mathtt{Mammal}$ derivation. An application that uses this ontology to retrieve mammals would then erroneously exclude giraffes. This issue can be caught by a simple automated test to check whether $\mathtt{Giraffe\sqsubseteq Mammal}$ is still entailed. Superficially, it may seem like this problem can be solved by just adding those axioms directly to the ontology. However, adding such axioms introduces a lot of redundancy, making modification of the ontology more difficult. Adding only a test instead ensures correctness without bloating the ontology. In addition, if it is specified as a test and documented as such in a testing environment, one can easily repeatedly re-run tests.

Tests may also be used outside an automated test suite in order to explore and understand an ontology. For example, an author might be assessing an ontology of animals for reuse and wants to verify that $\mathtt{Giraffe}\sqsubseteq\mathtt{Mammal}$ is entailed in the ontology. The author can simply create a corresponding temporary test and observe the result, saving the time it would take to browse the inferred class hierarchy in an ontology development environment such as Protégé.

A similar approach can be employed when developing a new ontology: create a temporary test to determine whether the axiom i) is already entailed, ii) would result in a contradiction or unsatisfiable class if it were to be added to the ontology, or iii) can be added safely. For instance, one may wish to check whether one can add the domain declaration $\exists\mathtt{eats}^{-}.\top\sqsubseteq\mathtt{Animal}$ to the ontology such that it would not cause itself or another class to become unsatisfiable. In this case, it would be just executing this one TDD test for this axiom. Compare that with laborious and time-consuming alternatives: 1) browsing to the object property declarations and/or clicking through all class axioms of classes that are not animals and manually inspect each one on whether they happen to participate in an axiom involving $\mathtt{eats}$, or 2) the standard approach of adding an axiom, running the reasoner, and then observing the consequences. Option 1 may be feasible for a small toy ontology consisting of a few classes, but not for typical ontologies, let alone large ontologies, and one probably still adds option 2 to it. Option 2 involves reclassification, which may be slow, and which a TDD test can avoid once the ontology is classified.

This gives us two broad use cases:

Declare many tests alongside an ontology and evaluate them all together in order to demonstrate quality or detect regressions. 2. 2.

Evaluate temporary tests as needed in order to explore an ontology or predict the consequences of adding a new axiom.

To satisfy both of these, a hard requirement is that tests must not reclassify the ontology, and they must produce results that identify the consequences of adding an axiom.

2.2 Tests within the development process

The previous examples assume there is an axiom to add, but that has to come from some place and more activities are going on when testing. We identified three scenarios for the former and illustrated an informal lifecycle process for the latter in [15]. For instance, i) the axiom may be a formalisation of a competency question formulated by domain experts, ii) a modeller may work with a template for axioms, a spreadsheet or an ontology design pattern for one or more axioms, or iii) a knowledge engineer may know already which axiom to add. Resolving this aspect is beyond the current scope, as even within the TDD part, multiple options already are possible even with the straightforward fail-pass character. This is illustrated in Figure 1 for a more detailed version of the TDD test cycle within the larger TDD lifecycle for the simplest base case. TDD tests for ontologies currently only pass or fail, but, cf. TDD tests in software development, one has to deal with classifying the ontology and handling any inconsistency or unsatisfiable classes as well. The reason that that classification step is present is because when a test evaluates to a ‘fail’, one does not know whether that axiom is absent just because the knowledge is missing or because it would result to an inconsistency if it were to be added, so one would have to check that anyway after all. Ideally, one would want to know before any edit why the axiom is not in the ontology, so as to be better informed about the next step(s) to take.

3 Related works

We briefly outline the few ontology testing implementations first and subsequently recap relevant aspects of automated reasoners for OWL ontologies.

Test-Driven Development for ontologies

Three TDD tools for ontologies have been proposed recently. TDDOnto is a Protégé plugin that allows test axioms to be specified in Protégé’s syntax and then uses the reasoner through the OWL API, for it was shown to be the most efficient technology examined [15]. Further, once the ontology is classified, one can run as many implemented tests as one wants without invoking a classification again, whereas the other tools require ontology classification with each axiom exploration; hence, by design, they are already less efficient than TDDOnto. The two other TDD approaches for ontology development use a subset of OWL or have a different scope. Tawny-OWL [8] is an ontology development framework implemented in Clojure (not a widely known and used programming language) and it provides predicate functions which query the reasoner that will return true/false. It can be used in conjunction with any testing framework, such as the built-in “clojure.test”. TDD from a domain expert perspective is explored with Scone [17], which is based on Cucumber [22]. It leverages controlled natural language and (computationally costly) mock individuals, as do the tests described in [10]. Like Tawny-OWL, Scone is also separated from the common ontology development tools. Tawny-OWL and Scone do not support testing object properties or data properties.

No attempt has been made to rigorously prove the correctness of the testing algorithms of TDDOnto, Tawny-OWL, or Scone. None of these three tools support the full range of axioms permitted in OWL 2. Most notably, in none of them it is possible to directly test axioms of the form $C\sqsubseteq D$ where $C$ is not a named class, such as $\forall R.E\sqsubseteq D$, i.e., where it is a class expression. In addition, all three tools give only limited information about the result of any test, being pass/fail in Tawny-OWL and Scone, with TDDOnto also reporting missing vocabulary. This hinders their usefulness as a means to explore an ontology or aid in development. Using TDD with arbitrary General Concept Inclusions (GCIs) and a more comprehensive testing model is possible with TDDonto2 that has been introduced recently [19], but the paper did not cover technical details and evaluation.

There are several related works that move in the direction of Agile ontology development [3, 4], such as the pay-as-you-go approach for Ontology-Based Data Access [23] and the axiom-based possible world explorer [9], a proposal for an ontology testing framework for requirements verification and validation [24], and the ‘preparation’ step of going from competency questions to computing TDD tests [25]. They have an affinity with TDD insofar as that they could integrate with, or even rely on, a well-functioning, reliable, implementation of TDD for ontology authoring, and they predominantly rely on SPARQL queries rather than an OWL reasoner.

Automated reasoners

There are numerous OWL 2-compliant automated reasoners that employ several mechanisms to handle OWL files; e.g., the OWL API [26], OWLlink [27] (a Java library that defines a widely-supported standard interface for reasoners), and OWL-BGP [28] (a Java library that implements SPARQL). OWL-BGP introduces an efficiency overhead which is not present in OWL API [15] and OWLlink specifies a protocol for communication between distributed components (availing of the OWL API) whose scenario is thus orthogonal. Therefore we consider only using the reasoner directly through the OWL API and its functionality.

Performance evaluation of TDDOnto found that implementations that introduced temporary “mock” individuals were substantially slower than all others [15]. The cause was not explicitly identified, but it is likely due to the need for reclassification of the ontology to include the new assertions. As stipulated in Section 1, reclassification is undesirable and therefore that approach is not appropriate when it can be avoided. Instead, after the ontology is initially classified, one should be able to test by making queries, which are assumed to be acceptably efficient. The OWL API reasoner interface specifies a “convenience” method named isEntailed that accepts any axiom and returns a Boolean indicating whether or not that axiom is entailed. However, it is not mandatory for reasoners to implement this method, and only at most half do so222There are 73 reasoners listed at http://owl.cs.manchester.ac.uk/tools/list-of-reasoners/ of which 38 showed evidence of maintenance since 2012, and of those, 19 reasoners list support for entailment.. We therefore do not want to rely on its use. Therefore, we will use other methods available.

4 Foundations of TDD for ontologies

After a few preliminaries, we introduce the model for testing ontologies and subsequently present a selection of the algorithms.

4.1 Preliminaries

We begin with two prerequisite definitions, and then identify relevant reasoner methods and their returned values.

Definition 1** (Ontology language $\mathcal{L}$).**

$\mathcal{L}$* is the language of ontology $O$ with language specification adhering to the OWL 2 standard [18], which has classes denoted $C,D\in V_{C}$ where $C$ or $D$ may be a named class or a class expression, object properties $OP\in V_{OP}$, individuals $I\in V_{I}$, and axioms $A\in O_{A}$ that adhere to those permitted in OWL 2 DL.*

Definition 2** (Signatures).**

The signature $\Sigma(A)$ of an axiom $A\in O_{A}$ in ontology $O$ represented in $\mathcal{L}$, is the set of all symbols in the axiom. Further, there is a class signature $\Sigma_{C}(A)\subseteq\Sigma(A)$, an object property signature $\Sigma_{\mathit{OP}}(A)\subseteq\Sigma(A)$ and an individual signature $\Sigma_{I}(A)\subseteq\Sigma(A)$.

Definition 3** (OWL reasoner methods).**

*Let $O$ be the ontology under test, $\{C,...\}\in V_{C}$ are class expressions; $N\in V_{C}$ is a named class; $a,b\in V_{I}$; and $R\in V_{OP}$. The following methods are available from the reasoner:

\begin{array}[]{rcl}&&\textsc{isSatisfiable}(C)\iff O\nvdash C\sqsubseteq\bot\\ &&\textsc{getSubClasses}(C)=\{N\in\Sigma_{C}(O)\mid O\vdash N\sqsubseteq C\}\\ &&\textsc{getInstances}(C)=\{a\in\Sigma_{I}(O)\mid O\vdash a:C\}\\ &&\textsc{getTypes}(a)=\{N\in\Sigma_{C}(O)\mid O\vdash a:N\}\\ &&\textsc{getSameIndividuals}(a)=\{b\in\Sigma_{I}(O)\mid O\vdash b\equiv a\}\\ &&\textsc{getDifferentIndividuals}(a)=\{b\in\Sigma_{I}(O)\mid O\vdash b\not\equiv a\}\end{array}*

Note that getSubClasses returns the union of equivalent classes and strict subclasses.

4.2 A model of testing ontologies

In order to rigorously examine any testing algorithms, we need a formal description of what it means to test an axiom against an ontology333Observe that the scope is testing axioms, not a broad informal meaning of ‘testing’ that also would include, say, checking for naming conventions.. In line with the use cases identified in Section 2, we define the possible test results. Instead of the underspecified three possible statuses of the existing tools (pass/fail/unknown), we specify seven cases that include the three existing ones and, principally, refine the ‘fail’ cases. They are listed in order from most grave failure to pass.

• •

Ontology already inconsistent. That is, $O\vdash\top\sqsubseteq\bot$. The reasoner cannot meaningfully respond to queries, so no claims can be made about the axiom.

• •

Ontology already incoherent There is at least one $C\in V_{C}$ such that $C\sqsubseteq\bot$.

• •

Missing entity in axiom. By Definition 2, we have $\Sigma(A)\nsubseteq\Sigma(O)$.

• •

Axiom causes inconsistency. If the axiom were to be added to $O$, then it would cause it to become inconsistent, i.e., $O\cup A\vdash\bot$.

• •

Axiom causes incoherence. If the axiom were to be added to $O$, it would cause at least one named class to become unsatisfiable.

• •

Axiom absent. The axiom is not entailed by the ontology ($O\nvdash A$) and can be added without negative consequences.

• •

Axiom entailed. The axiom is already entailed by the ontology ($O\vdash A$).

In the context of TDD, only “Axiom entailed” is a pass; all the others are test failures. The first two possible failures apply to the entire suite of tests rather than to any one, so they should be checked only once as preconditions before evaluating any tests. Therefore, we do not consider them in any of the algorithms in Section 4.3. Similarly, the missing entities case can be a simple check at the start of each test which does not affect how it is otherwise evaluated. Since there is no ambiguity, we henceforth abbreviate the remaining cases to “inconsistent”, “incoherent”, “absent”, and “entailed”. This leads to the following formal definition of the testing model.

Definition 4** (Model for testing).**

Given a consistent and coherent ontology $O$, and an axiom $A$ s.t. $\Sigma(A)\subseteq\Sigma(O)$, i.e.,

[TABLE]

then the result of testing $A$ against $O$ is:

[TABLE]

The resultant values are ordered according to graveness of failure: $\mathrm{entailed}<\mathrm{absent}<\mathrm{incoherent}<\mathrm{inconsistent}$. One could add nicer labels in a user interface, such as “this axiom is redundant” for “entailed”, but such considerations are outside the current scope.

4.3 Algorithms and analysis

We now introduce the algorithms and analyses, in the context of an ontology $O$, which cover the most used types of axioms for (complex) classes, individuals, and RBox axioms that can be expressed as class axioms (domain and range axioms, and functional and local reflexivity and their inverses)444currently not supported: 1) entity declarations and datatype definitions because they cannot meaningfully be tested, 2) $\mathrm{HasKey}$ axioms, for they are hardly used due to unusual semantics, 3) RBox axiom types other than listed above, because it is complex to detect inconsistencies [29] and they require non-standard reasoning services.. In the interest of space and readability, a selection of the algorithms and proofs is presented in this section, which is based on importance, novelty, and to demonstrate the approach to the algorithms and proofs. The remaining ones follow the same pattern and are explained briefly in the text; the complete set of algorithms and their proofs can be found in a technical report [30].

Overall, this leaves four generic class axioms, three assertions, and six object property axioms. Each algorithm is named according to the axiom it tests, as written in OWL 2 functional syntax, prepended with “test”. For example, the algorithm for testing $\mathrm{SubClassOf}$ axioms is named testSubClassOf.

4.3.1 Class axioms

In the class axioms permitted by OWL 2 DL, all arguments may be arbitrary class expressions, not just named classes, except for $\mathrm{DisjointUnion}(N,C_{1},\ldots,C_{n})$ in which $N$ must be a named class. Consequently, to determine if $C\sqsubseteq D$ holds, it is not sufficient to check if $C\in\textsc{getSubClasses}(D)$, because $C$ will not occur in this set if it is not a named class. To resolve this, we build class expressions from the arguments and query them for satisfiability and instances.

To test such GCIs, we introduce Algorithm 1, which tests subsumption of class expressions and we will show its correctness.

Lemma 1**.**

For any set of axioms $O$ and class expressions $C$ and $D$, $O\>\vdash\>C\sqsubseteq D\;\iff\;O\>\vdash\>C\sqcap\neg D\sqsubseteq\bot$.

This is well known and therefore the proof is not included.

Proposition 1**.**

testSubClassOf* is sound and complete for entailment. That is, $\textsc{testSubClassOf}(C,D)=\mathrm{entailed}\implies\operatorname{test}_{O}(C\sqsubseteq D)=\mathrm{entailed}$ and $\operatorname{test}_{O}(C\sqsubseteq D)=\mathrm{entailed}\implies\textsc{testSubClassOf}(C,D)=\mathrm{entailed}$.*

Proof.

Soundness Algorithm 1 can only return entailed at line 10, so the three if-conditions must all be false. So

[TABLE]

Now suppose $O\nvdash C\sqsubseteq D$. By Lemma 1, $O\nvdash C\sqcap\neg D\sqsubseteq\bot$. In other words, $C\sqcap\neg D$ is satisfiable, which contradicts the last term of Eq. 1. Hence the supposition is false, so $O\vdash C\sqsubseteq D\iff{}\operatorname{test}_{O}(C\sqsubseteq D)=\mathrm{entailed}$.

Completeness The algorithm returns entailed if Eq. 1 holds (see Soundness). From $\operatorname{test}_{O}(C\sqsubseteq D)=\mathrm{entailed}$ we have that $O\vdash C\sqsubseteq D$, and by Lemma 1, $O\vdash C\sqcap\neg D\sqsubseteq\bot$ so the last term of the equation is true. Since $C\sqcap\neg D$ is unsatisfiable, by the coherence precondition it has no named subclasses, and by the consistency precondition it has no instances. Therefore the first and second terms of the equation are also true. Therefore Eq. 1 holds, and so the algorithm returns entailed. ∎

Proposition 2**.**

testSubClassOf* is sound and complete w.r.t. inconsistency.*

Proof.

Soundness Algorithm 1 can only return inconsistent at line 4, so the first if-condition holds, so $\textsc{getInstances}(C\sqcap\neg D)\neq\emptyset$, which means there exists an individual $a$ such that $a:C\sqcap\neg D\iff{}a:C\;\land\;a:\neg D$. Under $O\cup(C\sqsubseteq D)$ it follows also that $a:D$, which is a contradiction, so $O\cup(C\sqsubseteq D)\vdash\bot\iff{}\operatorname{test}_{O}(C\sqsubseteq D)=\mathrm{inconsistent}$.

Completeness We have that $O$ is consistent, so it has an interpretation, but $O\cup(C\sqsubseteq D)$ is inconsistent, so it has no interpretations. Suppose the algorithm does not return inconsistent. Then it must be that $\textsc{getInstances}(C\sqcap\neg D)=\emptyset$. But in this case there exists an interpretation $\mathcal{I}$ which models both $O$ and $O\cup(C\sqsubseteq D)$. Let $\mathcal{I}$ be the interpretation of $O$ with the smallest domain. This means that the interpretation of any class $E$ only contains elements which correspond to individuals which must be in that class: $E^{\mathcal{I}}=\{x\in\Delta^{\mathcal{I}}\mid(\exists a\in\Sigma_{I}(O))\;a:E\land a^{\mathcal{I}}=x\}$. This clearly still models $O$ because every individual is still in all classes it is entailed to be in. Under the supposition, we have that $(C\sqcap\neg D)^{\mathcal{I}}=\emptyset$. So for any individual $a$, $a:\neg(C\sqcap\neg D)$. Letting $a:C$, then $a:\neg(C\sqcap\neg D)\implies{}a:\neg C\sqcup D\implies{}a:D$. From the construction of $\mathcal{I}$, this means that $C^{\mathcal{I}}\subseteq D^{\mathcal{I}}\implies{}\mathcal{I}\vDash C\sqsubseteq D$. So $\mathcal{I}$ also models $O\cup(C\sqsubseteq D)$. This contradicts the initial condition that $O\cup(C\sqsubseteq D)$ is inconsistent, so the supposition must be false, and therefore the algorithm returns inconsistent. ∎

Proposition 3**.**

testSubClassOf* is sound and complete w.r.t. incoherence.*

Proof.

Soundness Algorithm 1 can only return incoherent at line 6, so the first if-condition must be false and the second true. So $\textsc{getInstances}(C\sqcap\neg D)=\emptyset{}\land\textsc{getSubClasses}(C\sqcap\neg D)\neq\emptyset$ Therefore, by the second term, there exists some named class $N\in\Sigma_{C}(O)$ such that $N\sqsubseteq C\sqcap\neg D$. By the contrapositive of Proposition 2, $O\cup(C\sqsubseteq D)$ is consistent, so by Lemma 1,

[TABLE]

Completeness If $C\sqsubseteq D$ is added to $O$, then the only classes that are affected are $C$ and its subclasses. Consider a named class $N\sqsubseteq C$. If $O\nvdash N\sqsubseteq\neg D$ then it is possible that any element in the $N^{\mathcal{I}}$ is also in $D^{\mathcal{I}}$ and thus it is possible that $N\sqsubseteq D$. If this is true for all such $N$, then they are all satisfiable in $O\cup(C\sqsubseteq D)$ which is therefore coherent, so it must not be true for at least one $N$. That is,

[TABLE]

From the contrapositive of Proposition 2 we have that the first if-condition is false, and we have shown that the second if-condition is true, so the algorithm returns incoherent. ∎

Theorem 1**.**

testSubClassOf* is correct and terminating.*

Proof.

It has been shown that Algorithm 1 is sound and complete for entailment, inconsistency, and incoherence, and the result is absent when it is not one of these other three. Therefore the algorithm returns the correct result in all cases. Termination is trivial, since the algorithm contains no loops or recursion. ∎

Note that a test for local reflexivity can avail of the same algorithm, with testSubClassOf($\top$,ObjectHasSelf(R)) and likewise for irreflexivity, and it holds similarly for functional as testSubClassOf($\top$,ObjectMaxCardinality(1,R)), and likewise for inverse functional, and object property domain and range (see [30] for details).

Testing for equivalent classes is done with a testEquivalentClasses function. The algorithm is correct and terminating, which follows directly from its specification: it iterates through testSubClassOf in a nested for-loop for classes $C_{1},...,C_{n}$ (with $n\geq 2$), and given that testSubClassOf is correct and terminating, then testEquivalentClasses so is the former (see [30] for details).

Algorithm 2 for disjoint classes is sound and complete for entailment, inconsistency, and incoherence, following largely the proofs of testSubClassOf that it uses within its for-loop, and therefore not included here (These proofs are available in the online technical report [30]). Likewise, Algorithm 3 for disjoint union is correct and terminating, relying on the previous results for the test for equivalent classes and for disjoint classes.

4.3.2 Assertions

Observe that adding an assertion does not affect satisfiability of classes, so $O$ cannot become incoherent. We take this as given for all axioms tested in this section. Algorithm 4 tests equivalence and Algorithm 5 difference of individuals. When an algorithm accepts $n$ individuals $\{a_{1},\ldots,a_{n}\}$ as arguments, we use the shorthand $\mathbf{a}$ for this set. We use the integer variable $i$ to iterate over the indices of individuals given as arguments, and the variables $\delta$ and $\gamma$ to temporarily store a set of individuals. Soundness and completeness proofs are available in the online technical report [30].

Finally, the algorithm for class assertions checks whether the individual is an instance of a class expression, using the getInstances(C) function.

5 Evaluation of TDD with TDDonto2

In order to evaluate the model of testing and the algorithms, we developed a Prot́egé plugin, named TDDonto2, that implements these TDD features, which will be introduced in Section 5.1. The actual evaluation will be presented afterwards. We carry out two ontology authoring tests: a quantitative approach to editing efficiency (Section 5.2) and an experiment with 25 novice modellers (Section 5.3).

5.1 The TDDonto2 Protégé plugin

TDDonto2 [19] is an updated version of TDDOnto [15] and is also a Protégé 5.x plugin that uses the OWL API, as that was shown to have resulted in the best performance cf the other techniques investigated [15, 31]. It implements the new model of testing for TDD and the new TDD algorithms and also has a GUI to wrap around it that incorporates some of the Protégé features, such as autocomplete and recognising whether a term is in the ontology’s vocabulary. The plugin can be added to any tab in Protégé, as desired555it thus still allows the modeller to browse to the standard interface with the classes, object properties, and individuals tabs and so forth to author ontologies.. An annotated screenshot of the plugin with several tests is included in Figure 2. Within the broader setting of an overarching TDD methodology for ontology authoring, it covers the aspects from entering the axiom (wherever it came from) until (but excluding) the refactoring step. Like its predecessor TDDOnto, one can type a single axiom and evaluate it directly, add several axioms and evaluate the whole a set of axioms or a selected subset thereof, and add a single axiom or a set of axioms to the ontology with a one-click operation. The different possible statuses for the test outcomes are colour-coded, where ‘no evaluation’ is left blank, ‘entailed’ is highlighted in green, and the other statuses are highlighted in red. Examples that illustrate the tool and a tutorial-style screencast are available from https://github.com/kierendavies/tddonto2, as well as the source code and jar file.

Note that TDDonto2 is a proof-of-concept tool primarily to test the workings of the algorithms and investigate in more depth what the best components of a full TDD methodology would be, i.e., it has not been subjected to a full software development lifecycle and it does not yet cover all steps in a TDD methodology (e.g., it does not consider refactoring). Nonetheless, it serves to evaluate the TDD with TDDonto2 to obtain first indications whether there is any benefit to a test-first approach already.

5.2 Editing efficiency evaluation

Ontology authoring efficiency consists of two components: the number of clicks and keystrokes one has to carry out even when one is familiar with the interface and the repeated invocation of the reasoner with as one extreme case the reasoner invocation after each edit and the other one only at the end of all editing operations. The former depends mainly on the axioms and the interface design of the ontology editor and the latter is an orthogonal dimension that can be added based on the number of edits.

Keystroke models are well-known in Human-Computer Interaction research as a non-invasive way to determine how long a task will take with the software, which has been extended over the years to determine correlations between typing & browsing speed and programming performance, the emotional state of a user, and biometric identification of users to detect hackers (see [32, 33] and references therein). For the editing efficiency evaluation of test-first TDD-based ontology authoring with TDDonto2 vs the test-last-based baseline, we are interested in task completion, and within that scope, to eliminate the noise of users so as to get a clear understanding of the ‘number of clicks and keystrokes’ component and the effects of reasoner invocation to determine the best performance in the most efficient possible situation. This then puts forward the hypothesis that:

• H1

Given a time allocation to clicks and keystrokes and automated reasoning (classification), the overall editing time is lower for TDD in TDDonto2 than the case of the ‘expert user’ and its test-last approach.

The ‘expert user’ is idealised as someone who is an experienced ontology engineer, which is someone who has a high familiarity with the widely-used Protégé 5.x tool and who never browses to the wrong place nor gets sidetracked in the authoring process. That said, the evaluation method described in the next section easily can be amended for another ontology editor and a ‘novice factor’ can be added666for instance, slower that average typing, more latency in scrolling and moving the mouse, and adding “Mental Operator” time penalties for switching contexts (including to the wrong ones, like the wrong tab), and keyboard/mouse input devices; see also [32, 33] for further possible variables and metrics..

5.2.1 Materials and methods

Methods

To falsify, or validate, the hypothesis, we need a systematic and plausible scenario that can handle axiom input changes. The first aspect to establish is the type of edits. We assume a typical ontology that is not too lightweight, yet also does not use all corner cases of OWL 2 DL axioms and data about common axioms in an ontology. This resulted in the following set of 10 types of axioms, where $C$ and $D$ are named classes, $R$ is a simple object property, $c1$ an individual777There are obviously variations, but the main point here is the principle of how to approach computing an editing efficiency and what variables are relevant.:

1. (i)

simple class subsumption $C\sqsubseteq D$ 2. (ii)

simple existential (all-some) or simple universal $C\sqsubseteq\exists R.D$ or $C\sqsubseteq\forall R.D$ 3. (iii)

simple disjointness $C\sqcap D\sqsubseteq\bot$ or $C\sqsubseteq\neg D$ 4. (iv)

domain $\exists R\sqsubseteq C$ 5. (v)

range $\exists R^{-}\sqsubseteq D$ 6. (vi)

instance declaration $c1:C$ 7. (vii)

qualified cardinality constraint $C\sqsubseteq\,\,\geq n\,\,R.D$ or $C\sqsubseteq\,\,\leq n\,\,R.D$ or $C\sqsubseteq\,\,=n\,\,R.D$ 8. (viii)

non-simple class on the left-hand side (lhs) (other than domain and range axiom) 9. (ix)

‘closure’ axiom $C\sqsubseteq\exists R.D\sqcap\forall R.D$ 10. (x)

arbitrary class expression on the rhs, conjunction/disjunction $C\sqsubseteq\exists R.D\sqcap\exists S.(E\sqcup F)$

Let us assume for now that these 10 axioms types are the only ones, so as to show the principle of the calculations.

The second dimension for the editing efficiency calculation is the presence/absence in the ontology of all the vocabulary elements used in the axioms, i.e., whether the terms have yet to be typed up or not. Third, whether all those vocabulary elements are top-level entities in the hierarchy (i.e., directly subsumed by owl:Thing and topObjectProperty) versus all entities are located at the leaves in the deepest hierarchy in the ontology. One then can compute three core variants: a) the lower bound with existing vocabulary + top-level elements, b) the upper bound with new vocabulary + down in the hierarchy, and c) an average case by taking the average characters/term and hierarchy depth for a typical online ontology or computed for the specific ontology under evaluation. We opt for the latter option, i.e., we take a quantitative approach where pre-selected axiom types are filled with average vocabulary from an existing ontology. To do this, we need the following variables:

• •

$a=|character_{term}|$ where term is the class name or object property name;

• •

$b=|clicks\,\,for\,\,hierarchy\,\,depth|$, be this the class or object property hierarchy;

• •

$c=|character_{term}|$ where term is the instance name;

• •

their respective averages, $\bar{a}$, $\bar{b}$, and $\bar{c}$, for some arbitrary ontology or for a specific ontology that is to be edited.

The number of clicks and keystrokes for a set of editing operations is then calculated as follows. Let $n$ be the number of axioms to add, which are of a form as one of the 10 above and categorised as such. Then for each axiom $A_{i}$ ($1\leq i\leq n$), one can calculate the minimum of clicks ($k_{i}^{min}$) and the maximum ($k_{i}^{max}$) so that the best case scenario is $\sum_{1}^{n}k_{i}^{min}$ and the worst case scenario is $\sum_{1}^{n}k_{i}^{max}$. Alternatively, one can select an option for each axiom that mirrors one’s habit or fix the option to systematically compare the same actions across ontologies. Because this is the first such evaluation, we choose the latter case.

Next, the way the number of clicks and keystrokes is calculated for each axiom specifically is as follows. We examined the editing efficiency calculations by using Protégé 5.2, which offers several ways of adding an axiom in most cases, under the assumption that the vocabulary is already present in the ontology. The first one for axiom of type (i) is included here and the others are listed in Appendix A:

1. (i)

click ‘Classes’ tab (= 1), then either:

1. (a)

drag class to position, if sufficiently nearby (existing classes) = 1 2. (b)

click class - click SubClass Of - in ‘class expression editor’ type classname - click ok (existing classes) = $3+a_{C}$ 3. (c)

click class - click SubClass Of - in ‘class hierarchy’ click as far down in the hierarchy as needed - select class - click ok (existing or new classes) = $4+b_{C}$

The corresponding one for TDDonto2 is $a_{C}$ SubClassOf: $a_{D}=a_{C}+11+a_{D}=11+a_{C}+a_{D}$.

Further, when typing the name of a vocabulary element or keyword, we consider the autocomplete feature. For a term, this varies by the size and naming, but for keywords (e.g., SubClassOf:) this is one character + a tab. To even this out, such an autocomplete is set on 4 keystrokes. For instance, the above calculation for $a_{C}$ SubClassOf: $a_{D}$ in TDDonto2 then becomes 4 + 4 + 4 = 12.

For the principal case to examine, we factor in time in two ways, with the first one being the determining one for falsifying hypothesis H1:

• •

reasoning time, by the worst case scenario for reasoning where after each axiom, the reasoner is invoked in the standard Protégé vs an “evaluate all” in TDDonto2.

• •

allocating 1 second to each click and 0.3 second to each keystroke, which is based on the typing speed average of 190-200 characters per minute (and permutations thereof; see below);

For small ontologies that classify fast, the time-per-click is expected to be the major factor, whereas for lager or complex ontologies, the reasoning time is expected to be the major factor. The reasoning time is determined by classifying the ontology once, and then the worst case (reasoner invocation after each axiom) is computed by multiplying it by 9 as approximation of the total cost for Protégé and twice for TDDonto2 (at the start and after adding all axioms).

Lastly, the principal case will be checked against several permutations to assess robustness of results. They are as follows: (1) the same setting of 1s click and 0.3s keystroke but without autocomplete, (2) without autocomplete but slower clicking (2s) and faster typing (0.25s/keystroke), and (3) the same setting but 8 keystrokes for autocomplete cf. 4.

Materials

We selected three actual ontologies to compute the average values and augmented it with three ‘mock’ ontology averages. The actual ontologies are the African Wildlife Ontology (AWO), the Pizza ontology, and DataMining Optimization Ontology (DMOP). The AWO is a very small tutorial ontology (31 classes, 5 object properties, and 56 logical axioms, in $\mathcal{SRI}$ (i.e., OWL 2 DL)), which is used in the ontology engineering course of one of the authors888http://www.meteck.org/teaching/OEbook/ontologies/ and it also will be used for the user evaluation in the next section. The Pizza ontology [34] is deemed well-known; it has 98 classes, 8 object properties, and 786 logical axioms, and is in $\mathcal{SHOIN}$ (OWL DL) expressiveness. DMOP [35] is medium-sized and complex (723 classes, 96 object properties, and 2425 logical axioms, in $\mathcal{SROIQ}(D)$, i.e., OWL 2 DL). It was chosen because of its size and complexity and because two of the authors were involved in its development and thus would be able to analyse some internals if the results would demand for it. In addition, we added three mock ontologies such that the parameters have different values from AWO, Pizza, and DMOP such that the effects of the variables’ values for reasoner time, length of names, and hierarchy depth can be examined further. Their relevant characteristics are included in Table 1. Given that we calculate with averages, the actual number of classes etc in M1-M3 do not matter, nor does the DL fragment or OWL species as indication of possible reasoning times, for that is also a fixed value for each mock ontology, which was set to be in-between Pizza’s insignificant classification time on the one end and DMOP’s 20 minutes at the other end. While there are larger ontologies with longer classification times, 20 minutes is already substantial for the authoring process and much longer than the overhead of the TDDonto2 tool to compute the outcome of the tests.

The classification times for AWO, Pizza, and DMOP were recorded on a MacBook Pro with 2.7 GHz Intel Core i5 and 16 GB memory.

5.2.2 Results

The aggregate results in editing efficiency time with and without reasoner is shown in Figure 4. In order to falsify or validate hypothesis H1 on test-last vs test-first, compare “Total Protégé - single edit reasoner” with “Total TDDonto2 - with reasoner”: it is obvious that test-first is somewhat to much faster than test-last, ranging from 13s in case of the AWO to a staggering 8430s with the DMOP. Thus, H1 is validated.

Comparing the first two data series (i.e., without reasoner) to the latter two (with reasoner), it is clear that the reasoner has most effect on overall editing time for the medium to large ontologies (DMOP and mock2). For the two small ontologies, AWO and Pizza, it turns out that the click & keystroke time is the major contributor to the overall time taken. AWO’s clicks amount to 75.9s in Protégé, so with a single classification being 0.81s, it only reaches a total of 83.19s for Protégé’s worst case (invoke reasoner after each edit), which is 68.4s and 70.02s, respectively for TDDonto2 with its two invocations of the reasoner.

While the test-first is a clear winner, the difference is small for very small ontologies and the interface itself seems to have a bigger impact. Therefore, let us disaggregate the click-and-keystroke time by type of axiom. Then, the results are still favourable for TDDonto2—i.e., less time—except for a small difference for axiom type (ix) and (x); see Figure 3. We carried out a statistical analysis for each axiom type, where the null-hypothesis is that there is no difference. As the TDDonto2 series’ data are not normally distributed, we used Wilcoxon for paired data, 2-tailed, and significance level 0.05. This was calculated over the data of the 4 scenarios pooled together so as not to cherry-pick (and 6 data points is insufficient for Wilcoxon). It was statistically significant for all axiom types except for type 5 (see online supplementary data).

There are several other noteworthy observations from this interface-only data. First, unlike with Protégé, the editing efficiency with TDDonto2 is immune to the vocabulary size names and hierarchy depths. This is partially thanks to the autocomplete feature. Second, there is an obvious gradient in the Protégé data. Given the three mock ontologies that varied class depth and vocabulary name size, it demonstrates that the hierarchy depth has the most negative effect, given mock2’s $b_{C}=12$ and lower vocabulary name size cf DMOP, yet having higher values overall, and cf. mock1 and mock3 that have similar results but only the low hierarchy depth value remained the same.

Note also that the values for axiom types (ix) and (x) are similar and now also invariant for Protégé, because of the possible options to add it to the ontology and the task execution scenario selected was that of the class expression editor rather than firing the steps for axiom type (ii) twice and related auxiliary clicks (see Appendix for details). That is, Protégé now also requires the user to type the characters of the axiom, which is similar to the TDDonto2 interface. Using their respective scenario (a) with clicking, then the differences with TDDonto2 are the largest, as shown in Figure 5, for it is a compounding effect adding up from the simpler axiom types. As they rely on axiom type (ii), which was already statistically significantly faster with TDDonto2, then so it is for axiom types (ix) and (x).

The comparison with other scenarios that vary click and keystroke times and autocomplete indicates robustness of results. That is, in the first alternate scenario (no autocomplete), Protégé has slightly lower values for most axiom types. In the second and third scenarios (slower clicks and less autocomplete, respectively), TDDonto2 has lower values for most axiom types and they show a similar pattern as in Figure 3 where axiom types (i)-(vii) are better for TDDonto2 and (ix)-(x) slightly better with Protégé (data not included). Thus, on the micro-level of the axiom, there is some editing difference between Protégé 5.2 and TDDonto2 for (very) small ontologies and an expert user, which becomes larger in favour of TDDonto2 for medium-sized and large ontologies. The latter is even more pronounced when taking into account the reasoner.

5.2.3 Discussion

It may be clear from the materials and methods section that there can be many parameters to assess editing efficiency, that, perhaps, detracts one from the core result of test-first vs. test-last with the reasoner—the former having a higher editing efficiency. For the interface interaction, we made choices that seem reasonable to us, such as the autocomplete feature and the average case only and, to some extent, which of the alternate interface interaction scenarios was selected to be included999e.g., for (ix) and (x), the clicking option could have been selected cf. the class expression editor, which resulted in the large difference with TDDonto2, as shown in Figure 5. Besides that the authors would choose the latter option over firing the steps for axiom type (ii) twice, the class expression editor is the only option in case of a disjunction on the right-hand side, and therefore the class expression editor had been selected upfront.. This serves the investigation into simulating a set of expert users in tool evaluations as well as teasing out parameters for non-human quantitative evaluations of ontology authoring tasks. In the case of the common Protégé 5.2 interface vs TDDonto2, the latter generally comes out favourably in several scenarios. This is especially so for medium and large-sized ontologies, which is further amplified with greater class hierarchy depth. In addition, if a ratio of axiom types would have been selected to be included in the calculations, rather than one of each, it would be mostly of type (ii), where TDDonto2 is distinctly more efficient, so this beneficial effect would thus be amplified further. Thus, ontologies within OWL 2 EL expressiveness characteristics (tailored to large, ‘simple’ TBoxes), such as a SNOMED CT, would benefit most from TDD and its axiom-level input in TDDonto2. This would be even more so for the scenario where new vocabulary has to be added, for the additional typing has a comparatively larger effect on the predominantly clicking-based Protégé interface with the existing vocabulary scenario we opted for. Finally, one may argue that most ontology developers are not expert efficient users and the editing efficiency calculations are artificial. Therefore, we shall return to this factor in the next section with the user evaluation.

The set-up has been lenient on the automated reasoner. This is because the aim was to obtain a general sense of any possible benefit in the authoring process, rather than aiming for improvements in the seconds. Practically, reasoning time may increase greatly as a result of adding an axiom. A quantitative approach would amount to randomised adding of axioms, so it would be an unpredictable effect. Reasoner performance per sé is not the scope of the paper, however, and therefore we deemed the approximation of 9x the baseline acceptable, rather than adding more than the baseline. Further, the mode of approximation taken is favourable for the test-last setting rather than TDD, for the former requires more often the invocation of the reasoner. That TDD emerged positively already suggests it will be even more so in praxis.

5.3 User study

The editing efficiency evaluation reported on in the previous section assumed an optimal user working on an average ontology. Here, we are evaluating actual novice users on a relatively small ontology. We run this user study to test the following claims:

• C1

Users will complete a larger part of the task when using the test-first TDDonto2 Protégé plugin than when using the test-last basic ontology editor interface of Protégé.

• C2

Users will make fewer mistakes when using the TDDonto2 Protégé plugin than when using the basic ontology editor interface.

• C3

Users will be able to complete the tasks in less time when using the TDDonto2 plugin than when using the basic ontology editor interface.

5.3.1 Materials and methods

The study included two general subtasks (assessing TBox axioms and ABox axioms) in one domain: African wildlife.

The subjects were master students studying computer science, who have basic knowledge of ontologies and semantic technologies. The users had been informed that the purpose of the user study was to evaluate using a tool for introducing knowledge into an ontology, a plugin to the Protégé editor TDDonto2, compared to using the existing Protégé interface without the plugin. The users were presented with a demo of TDDonto2101010accessible via https://github.com/kierendavies/tddonto2 and an ontology to be extended.

The general task was to assess the status of each axiom from the two sets (TBox axioms and ABox axioms) with respect to the ontology. The subjects could select the status from the following set: $\{$entailed, absent, incoherent, inconsistent$\}$, and they were provided textual definitions of each of the statuses. We measured what percent of the statuses of the axioms the subjects were able to assess within given time (completeness) and what percent of their assessments were correct (correctness).

The experiment procedure was as follows:

For the given set of axioms, follow these steps:

Register the current time (in the field provided).

• -

Select the status of the given axiom (“Entailed”, “Absent”, “Incoherent”, “Inconsistent”) for each axiom from the set,

• -

Enter the axioms for which you have marked “Absent” into the ontology and save the ontology file.

• -

Copy the ontology source file to the given field.

• -

Register the current time (in the field provided).

The experiment was designed as a within subject study. The subjects were divided into two groups corresponding to the attendees of two class labs. There were 25 subjects altogether: 13 in the first group (which we denote Group A), and 12 in the second group (which we denote Group B). They were assigned four tasks:

• •

Task $T1$: Testing the introduction of axioms regarding classes – without TDDonto2 plugin,

• •

Task $A1$: Testing the introduction of axioms regarding instances – without TDDonto2 plugin,

• •

Task $T2$: Testing the introduction of axioms regarding classes – with TDDonto2 plugin,

• •

Task $A2$: Testing the introduction of axioms regarding instances – with TDDonto2 plugin.

In Group A, the subjects first used the basic editor interface to complete the tasks and then TDDonto2. In Group B, the subjects first used TDDonto2 to complete the tasks and then the basic editor interface. The time to complete tasks $T1$ and $T2$ (TBox axioms) was limited to 20 minutes per each task, and the time to complete tasks $A1$ and $A2$ (ABox axioms) was limited to 5 minutes for each task. Furthermore, to avoid the issue of a transfer effect (i.e., not repeating the errors the second time the subjects do the task) we prepared two different but comparable axiom sets111111The tested axioms are at aforementioned URL (fn. 10).

5.3.2 Results and discussion

Table 2 shows the data and statistics on the results of the user study, corresponding to claims C1-C3, with a breakdown by groups and the interfaces used. Regarding claim C1 (see Table 2a), we can see that, on average, more subjects completed the tasks when using TDDonto2 (94% of task completeness versus 90% regarding the basic interface), which supports our claim. Regarding claim C2 (see Table 2b), relatively more subjects (88%) correctly completed the tasks when using TDDonto2, while the percent of correct answers was lower in case of using the basic editor (62%), which supports claim C2. Regarding claim C3 (see Table 2c), we can see that the average time of completing the task was more than 2 times shorter when using TDDonto2 than when using the basic editor, more precisely it constituted 44% of the time of completing the tasks when using the basic editor, which supports our claim C3.

Figure 6 shows the comparison of correctness of completing the tasks when using the basic interface, and TDDonto2, disaggregated by the types of axiom statuses.

We have performed a t-test for statistical significance of correctness percentage results. We generated two sets for three settings (per TBox, per ABox, and per TBox plus ABox): a set with overall correctness percentage per each user per each task for the basic interface and a set with overall correctness percentage per each user per each task for TDDonto2. The null hypothesis was that relevant two sets of correctness percentages (the results for the basic interface and for TDDonto2) had identical mean (expected) values. The results for the TBox experiments are: statistic=-3.3497, $p$=0.0015. The results for the ABox experiments are: statistic=-2.8108, $p$=0.0072. The overall result (TBox plus ABox) is as follows: statistic=-4.3337, $p$=3.55e-05. Since the value of $p$ in all the cases is below 0.05, we reject the null hypothesis and conclude that the difference in the correctness results between the basic interface version of the experiments and the version when the subjects used TDDonto2 is statistically significant.

Finally, note that in the editing efficiency evaluation, editing in the AWO was about the same for Protégé and TDDonto2, for an assumed efficient expert user. Yet, these results with actual users demonstrate clearly that in praxis one can already observe TDD benefits even in these settings of small ontologies already.

6 Discussion

TDD as test-first approach to ontology authoring has been shown to be theoretically and technologically a viable option, and the first user study indicated that it is also beneficial for the authoring process from a user perspective. This clearly can be embedded in a broader process of ontology engineering, as the proposed lifecycle in [15] already suggested. This can be extended further to also include goal or behaviour-driven development, which Scone [17] aims at, and conversion of competency questions into axioms that would feed into the technical TDD component presented in this paper by, e.g., linking it to [25]. The TDD component of regression testing—verifying past tests still pass—also may be an avenue for future works. Overall, these additions change the general flow of a TDD test of Figure 1 into the one shown in Figure 7. Also in this case, more scenarios are possible than shown, so as not to obscure the general idea. For instance, after resolving conflicts when a precondition fails, one may not want the axiom in the ontology anymore. The current version of the TDDonto2 tool can cater for these variants, but it has no explicit interface features for them at present and it is left to the modeller’s decisions.

Concerning theoretical and feature advances, the algorithms presented in Section 4.3 are the first ones with a broad coverage of OWL 2 language features, superseding those presented in all related work [15, 17, 8] especially on GCIs and the ABox, and also proving correctness of encoding. In addition, the model of testing axioms goes beyond the pass/fail/unknown and reporting missing vocabulary of the related work (which it does, too), by providing other possible outcomes that clarify what sort of a ‘fail’ it is. This is a distinct feature for the setting of ontologies cf. software engineering, where a fail simply means “not present, to implement”: the ‘fail/not present’ may be because of absence due to lack of coverage, indeed, but also may be because adding it would cause inconsistency or incoherence, which is something one would want to know to determine the next step in the ontology authoring process. That is, unlike in software development, a ‘fail’ does not necessarily imply ‘to add’.

We did make certain design decisions for this TDD that one may want to experiment with aside from the choice of technology121212Note that alternatives to using the reasoner directly have been investigated, notably BGP with SPARQL-OWL [28] and instance-based approach with mock objects, but exploiting the OWL reasoner turned out to be the fastest [15, 31].. For instance, once isEntailed is implemented by most or all reasoners, one could choose to update some of the algorithms accordingly. Also, one may also want to relax the coherency precondition. In our Model for Testing specification (Definition 4), we sided with the somewhat ‘hardline’ approach from a logician’s viewpoint—a consistent theory, and every element satisfiable—compared to a possible tolerance for unsatisfiable classes at some point in the authoring stage. Anecdotally, we have seen behaviour along the line of “yes, I know x and y are inconsistent but I do not want to deal with them now”. Within the TDD scope, it would be preferable to remove them from the ontology, and add them at least temporarily as TDD test in the test set. This possibility eliminates issues with cascading unsatisfiable classes, yet not somehow losing that knowledge that with the test specification has become more easily examinable and thus resolvable.

The overall time of authoring an ontology is reduced thanks to not invoking the reasoner for each edit, which many a developer does [13], yet still being able to evaluate what the outcome would be if the axiom were to be added to the ontology. It does not reduce the reasoning time for the first classification, nor after actually having modified the ontology. Such efficiency improvements are reasoner improvements (e.g., using incremental reasoning), whereas here we focus on authoring improvements.

7 Conclusions

The novel test-driven development algorithms introduced in this paper fill a gap in rigour and coverage of both types of axioms that can be tested with a test-first approach and it provides more feedback to the modeller by means of its model of testing. The evaluation of this test-driven development in TDDonto2 with a novel human-independent assessment approach for editing efficiency demonstrated that it is faster than the typical ontology authoring interface (Protégé 5.2) to some extent for smaller ontologies and even more so for medium to large ontologies with a stylised expert modeller, especially when automated reasoning is factored into the authoring process. Further, the user evaluation demonstrated that it is also more effective in task completion, time, and correctness (quality) for smaller ontologies and relative novice users. Thus, TDD’s test-first approach with TDDonto2 is more effective than the common test-last authoring approach with Protégé.

The results demonstrate promise of test-driven development as an ontology development methodology. To turn it in a complete methodology, other components can be investigated, such as the refactoring step and the interaction with competency questions.

Acknowledgments

This work was partly supported by the Polish National Science Center (Grant No 2014/13/D/ST6/02076).

Appendix

The calculations of the interface clicks for Protégé 5.2 are as follows. Given that several options are typically possible, we select one, which is indicated with an asterisk at the end of the option. noting that it offers several ways of adding an axiom in most cases.

1. (i)

click ‘Classes’ tab (= 1), then either:

1. (a)

drag class to position, if sufficiently nearby (existing classes) = 1 2. (b)

click class - click SubClass Of - in ‘class expression editor’ type classname - click ok (existing classes) = $3+a_{C}$ 3. (c)

click class - click SubClass Of - in ‘class hierarchy’ click as far down in the hierarchy as needed - select class - click ok (existing or new classes) = $4+b_{C}$ [∗] 2. (ii)

click ‘Classes’ tab (=1), then either:

1. (a)

click class - click SubClass Of - in ‘class expression editor’ type “R some/only D” - click ok = $3+a_{OP}+4+a_{D}$ 2. (b)

click class - click SubClass Of - in ‘Object restriction creator’ click as far down in the property hierarchy as needed - select property - in ‘Object restriction creator’ click as far down in the restriction filler as needed - select class - click restriction type some - click ok = $5+b_{OP}+b_{C}$ [∗] 3. (iii)

click ‘Classes’ tab (=1), then either:

1. (a)

click class - click Disjoint With - in ‘class expression editor’ type classname - click ok (existing classes) = $3+a_{C}$ 2. (b)

click class - click Disjoint With - in ‘class hierarchy’ click as far down in the hierarchy as needed - select class - click ok = $4+b_{C}$ [∗] 4. (iv)

click ‘Object properties’ tab (=1), then either:

1. (a)

click property - click Domain - in ‘class expression editor’ type classname - click ok = $3+a_{C}$ 2. (b)

click property - click Domain - in ‘class hierarchy’ click as far down in the hierarchy as needed - select class - click ok = $4+b_{C}$ [∗] 5. (v)

has the same processes as for (iv). 6. (vi)

click ‘Individuals by class’ tab (=1), then either:

1. (a)

click Types - in ‘class expression editor’ type classname - click ok = $2+a_{C}$ 2. (b)

click Types - in ‘class hierarchy’ click as far down in the hierarchy as needed - select class - click ok = $3+b_{C}$ [∗] 3. (c)

click ‘Classes’ tab (=1), then click Instances - click instance - click ok = 3 7. (vii)

click ‘Classes’ tab (=1), then either:

1. (a)

click class - click SubClass Of - in ‘class expression editor’ type “R min x D” - click ok = $3+a_{R}+3+1+a_{D}=7+a_{R}+a_{D}$ 2. (b)

click class - click SubClass Of - in ‘Object restriction creator’ click as far down in the property hierarchy as needed - select property - in ‘Object restriction creator- click as far down in the restriction filler as needed - select class - click restriction type - click/type cardinality - click ok = $7+b_{OP}+b_{C}$ [∗] 8. (viii)

click ‘Active ontology’ tab (=1), then

1. (a)

click ‘General class axioms’ - click add - type the entire GCI - click ok = 3 + GCI 9. (ix)

click ‘Classes’ tab (=1), then either:

1. (a)

execute the clicks for axiom type (ii) twice 2. (b)

click add - click class expression editor, and type: some $a_{R}$ $a_{D}$ and only $a_{R}$ $a_{D}$ = 2+ 4 + $a_{R}$ + $a_{C}$ + 3 + 4 + $a_{R}$ + $a_{D}$ = 13 + $2*a_{R}$ + $2*a_{D}$ [∗] 10. (x)

click ‘Classes’ tab (=1), then either:

1. (a)

execute the clicks for axiom type (ii), then click add - click class expression editor and type: some $a_{S}$ ($a_{E}$ or $a_{F}$) = [axiom type (ii) clicks] + 2 + 4 + $a_{S}$ + 1+ $a_{E}$ + 2 + $a_{F}$ + 1 = [axiom type (ii) clicks] + 10 + $a_{S}$ + $a_{E}$ + $a_{F}$ 2. (b)

click add - click class expression editor, and type: some $a_{R}$ $a_{D}$ and some $a_{S}$ ($a_{E}$ or $a_{F}$) = 2 + 4 + $a_{R}$ + $a_{D}$ + 3 + 4 + $a_{S}$ + 1+ $a_{E}$ + 2 + $a_{F}$ + 1 = 16 + $a_{R}$ + $a_{D}$ + $a_{S}$ + $a_{E}$ + $a_{F}$ [∗]

The clicks formulae with TDDonto2 are as follows. In the TDDonto2 plugin, one only adds full GCIs/assertions and then the user has to click “Add” (=1), and then, for the 10 axiom types, in the same order:

1. (i)

$a_{C}$ SubClassOf: $a_{D}=a_{C}+11+a_{D}=11+a_{C}+a_{D}$ 2. (ii)

$a_{C}$ SubClassOf: some $a_{R}$ $a_{D}=a_{C}+11+4+a_{R}+a_{D}=15+a_{C}+a_{R}+a_{D}$ 3. (iii)

$a_{C}$ and $D$ SubClassOf: owl:nothing or $a_{C}$ SubClassOf: not $a_{D}=a_{C}+3+a_{D}+11+11=a_{C}+a_{D}+22$ or $a_{C}+11+3+a_{D}=a_{C}+a_{D}+14$ 4. (iv)

some $a_{R}$ SubClassOf: $a_{C}=4+a_{R}+11+a_{C}=15+a_{R}+a_{C}$ 5. (v)

some (inverse($a_{R}$) SubClassOf: $a_{D}=4+9+a_{R}+11+a_{D}=24+a_{R}+a_{D}$ 6. (vi)

c1 Type: C = $c+5+a_{C}$ 7. (vii)

$a_{C}$ SubClassOf: $a_{R}$ min n $a_{D}=a_{C}+11+a_{R}+4+a_{D}=a_{C}+15+a_{R}+a_{D}$ 8. (viii)

non-simple class on the lhs (other than domain and range axiom) = can be anything 9. (ix)

$a_{C}$ SubClassOf: some $a_{R}$ $a_{D}$ and only $a_{R}$ $a_{D}$ = $a_{C}+11+4+a_{R}+a_{D}+3+4+a_{R}+a_{D}=22+a_{C}+2*a_{R}+2*a_{D}$ 10. (x)

$a_{C}$ SubClassOf: some $a_{R}$ $a_{D}$ and some $a_{S}$ ($a_{E}$ or $a_{F}$) = $a_{C}$ + 11 + 4 + $a_{R}$ + $a_{D}$ + 3 + 4 + $a_{S}$ + 1+ $a_{E}$ + 2 + $a_{F}$ + 1 = 26 + $a_{C}$ + $a_{R}$ + $a_{D}$ + $a_{S}$ + $a_{E}$ + $a_{F}$.