Semi Automatic Construction of ShEx and SHACL Schemas

TL;DR

This paper introduces a semi-automatic approach combining an algorithm and interactive tools for constructing SHACL and ShEx schemas from RDF datasets, facilitating schema creation with expert input and validation features.

Contribution

It presents a novel method integrating automatic schema construction with interactive editing for efficient schema development.

Findings

Effective schema construction from sample nodes

Interactive workflow enhances schema accuracy

Combines algorithmic and manual schema refinement

Abstract

We present a method for the construction of SHACL or ShEx constraints for an existing RDF dataset. It has two components that are used conjointly: an algorithm for automatic schema construction, and an interactive workflow for editing the schema. The schema construction algorithm takes as input sets of sample nodes and constructs a shape constraint for every sample set. It can be parametrized by a schema pattern that defines structural requirements for the schema to be constructed. Schema patterns are also used to feed the algorithm with relevant information about the dataset coming from a domain expert or from some ontology. The interactive workflow provides useful information about the dataset, shows validation results w.r.t. the schema under construction, and offers schema editing operations that combined with the schema construction algorithm allow to build a complex ShEx or SHACL schema.

Figures

Equations

$$pr\text{:}⪯\mathsf{iri}⪯\mathsf{nonlit}⪯\mathsf{any}\mathsf{blank}⪯\mathsf{nonlit}X⪯\mathsf{lit}⪯\mathsf{any}$$

$$consensus(O,⪯,W,\pi ,t)=\bigvee \left\{o\mid {\text{acc}}_{\pi }(o)=a\right\}.$$

$${\pi }_{K,p}(n)=\{\begin{array}{ll}\{0;1\}& \text{ if }m=0\\ \{1;1\}& \text{ if }m=1\\ \{1;\ast \}& \text{ if }m>1\end{array}$$

$$props(P,N)=\text{predicates}(P)\cup \left\{q\in props(N)\mid match(P,q)\ne \perp \right\}.$$

Full text

1 Introduction

With the growth in volume and heterogeneity of the data available in the Linked Open Data Cloud, the interest in developing unsupervised or semi-supervised methods to make this information more accessible and easily exploitable keeps growing as well. For instance, RDF graph summarization [1] is an important research trend that aims at extracting compact and meaningful information from an RDF graph with a specific purpose in mind such as indexing, obtaining high level descriptions of a given dataset, graph visualization, query optimization, or extracting a schema or an ontology.

The Shapes Constraint Language (SHACL) [6] is the W3C standard for defining constraints for RDF graphs. The Shape Expressions Schemas (ShEx) language  [14] has a similar purpose but is based on a slightly different set of operators. Both proposals allow to define a set of named shapes that constrain the RDF nodes and describe their close neighbourhoods. A set of such constraints is called a schema and provides high level information about the structure and contents of an RDF dataset. Such schema is usually constructed with a particular application in mind. SHACL and ShEx schemas allow to increase RDF datasets usability, as they facilitate RDF validation, understanding and querying.

More recently, several authors have used techniques close to graph summarization in order to automatically produce a SHACL [12, 15, 10] or a ShEx [2, 16, 7, 4] schema. Nevertheless, shapes generated using noisy or incomplete data may contain incorrect data or imprecisions. Most of the times, the shapes should be reviewed and edited by domain experts in order to ensure they fit with the application requirements. The usability of tools letting a proper edition of those schemas has been identified as a challenge [8]. The relative youth of consolidated constraint languages for RDF data causes that this kind of tools has not been thoroughly investigated yet, but a list of desirable features has been proposed [3].

Contributions

In the current paper we describe the algorithms and the software components that are necessary for the development of an integrated, highly parametrizable and well automated tool that would allow to users with different levels of expertise to construct SHACL and ShEx schemas from data. As a first contribution, we define an algorithm that takes as input a set of sample nodes from the RDF dataset and outputs a so called uniform shape constraint that uses only a restricted number of features of ShEx and SHACL. The uniform constraint can either be satisfied by all the nodes in the sample, in which case it is the unique most specific such constraint. Or it can take into account possible noise in the data and be satisfied by most sample nodes except for a small number of them consistent with a predefined error threshold. As a next contribution, we introduce schema patterns as a general mechanism for guiding and tuning our schema construction algorithm. A schema pattern can be provided as input to the algorithm, in which case the latter can output a complex shapes schema with nested constraints and references between shapes. In particular, we show how to construct a schema pattern from an existing ontology so that the automatically constructed schema contains one shape for each class in the ontology and integrates the predicates' domain and range axioms. We finally describe an interactive workflow that allows to construct an arbitrarily complex schema with the help of appropriate schema edition operators, statistical information about the data, a comprehensive visualization of validation results, and integrated calls to the algorithm for automatic schema construction.

Most of the ideas presented here are already implemented in the open source java tool ShapeDesigner available at https://gitlab.inria.fr/jdusart/shexjapp.

Related work

RDF2Graph [2] performs structural graph summarization on knowledge bases that can be exported to RDF, OWL, XGMML or ShEx. By default, this tool associates shapes to classes, taking into account class hierarchies. However, some tuning is possible, allowing to define constraints for sets of nodes that do not belong to the same class. It is limited w.r.t. expressible cardinalities but considers incoming and outgoing links in the neighbourhood of nodes. The resulting summary can be exported to ShEx or visualized as a graphical network, but none of these options are intended to be edited or refined by the user.

RDF2Graph has been used as a base to perform shape construction in Wikidata [16], using a publicly available tool111https://tools.wmflabs.org/wd-shex-infer/ retrieved on 2019/04/11.. This tool allows to specify an initial custom set of focus nodes to be associated with a shape. It can relate other shapes to the initial one, being those shapes constructed w.r.t nodes of the same class. It ignores triples in reified models, thus the schema of the metadata associated to a given piece of knowledge in Wikidata is missed.

In [4] and [7] the same authors propose two tools for automatic generation of ShEx schemas, which can be partially translated to SHACL in [7]. The tool in [4] takes as input a set of classes of a Knowledge Base (KB) and produces a ShEx shape for each one computing the immediate neighbourhood of the instances of that class. It uses statistical methods to choose the neighbourhood constraints, being able to discard the infrequent constraints and providing statistical annotations about them. The approach presented in [7] is independent of classes and the resulting constraints go beyond the immediate neighbourhood, but it is more limited w.r.t. cardinalities and less parametrizable. None of those proposals define an interaction process, and they are both less parametrizable than our method w.r.t. the structure of the schemas to be constructed.

The tool presented in [12] detects sets of nodes of an RDF graph that match some patterns, then combines those patterns to construct more complex OWL 2 EL class expressions and axioms. The latter can be exported to SHACL. The node groupings used for defining a shape are not necessarily related to the same class nor selected by the user, but they are automatically detected by the mining algorithm.

ABSTAT [13] is an ontology-driven tool able to produce semantic profiles of a given Knowledge Base (KB). Those profiles gather statistics about type and cardinality of features in the immediate incoming or outgoing neighbourhood of a set of target nodes. In [15], ABSTAT's profiles are translated to SHACL shapes associated to the instances of some target class.

PaTyBRED [11] is a hybrid classifier method which relies on path and type relations for the detection of errors in KBs. In [10], the authors train decision trees using PaTyBRED to generate SHACL constraints. The core idea is to detect examples qualified as highly erroneous, turn them into logical conditions and use their negation as a constraint.

The novelties of our approach compared to those cited above are

(1) we include a strategy to integrate the user in the refinement of the automatically constructed schemas; and

(2) we introduce schema patterns as an expressive parametrization mechanism. This allows to feed the schema construction algorithm with useful information coming from an ontology, or with expert knowledge.

Organisation

In Sect. 2 we introduce a shapes schema language as an abstraction of ShEx and SHACL. In Sect. 3 we define uniform schemas as restriction of the language and explain the construction of the most specific and the largely accepted consensus uniform constraints. Schema patterns are defined in Sect. 4 and the interactive workflow in Sect. 5. We also discuss there the current status of the implementation of the method presented in this paper. We finally conclude and discuss future work in Sect. 6. This submission is accompanied by a long version extended with two appendices.

2 The Shape Constraint Language

We assume that the reader is familiar with RDF and has at least some basic understanding of the SHACL and ShEx constraint languages. We start by fixing notations related to RDF graphs then present the shapes constraint language.

Denote by $\mathsf{IRI}$ the set of IRIs, by $\mathsf{Lit}$ the set of literal values, and by $\mathsf{Blank}$ the set of blank nodes. Recall that an RDF graph $\mathrm{\mathbf{G}}$ is a set of triples in $(\mathsf{IRI}\cup\mathsf{Blank})\times\mathsf{IRI}\times(\mathsf{IRI}\cup\mathsf{Lit}\cup\mathsf{Blank})$. If $(s,p,o)$ is a triple, then $s$ is the subject, $p$ the predicate, and $o$ the object of the triple. A graph $\mathrm{\mathbf{G}}$ will be clear from the context all along the paper. The elements of $\mathsf{IRI}\cup\mathsf{Lit}\cup\mathsf{Blank}$ that appear in subject or object position of some triple of $\mathrm{\mathbf{G}}$ are called nodes, and the elements that appear in predicate position are called predicates or also properties. If $N$ is a set of nodes, we define $\mathit{props}(N)=\{p\mid\exists(s,p,o)\in\mathrm{\mathbf{G}}\textrm{ s.t. }s\in N\}$, i.e. the set of properties of the nodes in $N$. The neighbourhood of a node $n$ is the set of triples of $\mathrm{\mathbf{G}}$ with subject $n$. Node $n^{\prime}$ is called a $p$-neighbour of node $n$ if $(n,p,n^{\prime})$ is a triple in $\mathrm{\mathbf{G}}$.

Throughout the paper we use IRI prefixes without defining them and assume that different prefixes (e.g. p: and q:) always have different definitions.

SHACL [6] and ShEx [14] are the two main languages used for defining constraints for RDF graphs. See [9] for an introduction to these languages. They both use the notion of shape in order to describe or constraint the neighbourhood of nodes in RDF graphs, as illustrated in Example 1.

Example 1 (The shapes <Person> and <Date>)

The shape <Person> requires

1. an rdf:type link to foaf:Person;

2. zero or more owl:sameAs links to some IRI;

3. a foaf:name link and a foaf:familyName link to string literals; and

4. a bio:birth link to a xsd:gYear or a rdgr2:dateOfBirth link to a node that satisfies shape <Date>, but not both.

On the other hand, shape <Date> requires

1. an rdf:type link to time:Instant; and

2. an rdfs:label that is an integer value.

This is how the constraint for shape <Person> would be written in the shapes constraint language.

⬇

{ rdf:type [foaf:Person] {1;1}; owl:sameAs IRI {0;*} ;

foaf:name xsd:string {1;1}; foaf:familyName xsd:string {1;1} ;

( bio:birth xsd:gYear {1;1} | rdgr2:dateOfBirth @<Date> {1;1} ) }

In Appendix A we show how these shapes are expressed in ShEx and SHACL.

In Fig. 1 we provide some turtle content with a couple of nodes to be validated against the shape <Person>. :virginia conforms with <Person>, since it includes all the requested features regarding outgoing links. Shapes are open in the sense that they allow nodes to have outgoing links with predicates that are not mentioned in the shape, as for instance the property rdgr2:placeOfBirth of :virginia. :william does not conform, since the information about Williams' name is provided using schema:name instead of foaf:name and foaf:familyName. In addition, :william also violates the requirement that only one among bio:birth or rdgr2:dateOfBirth links should be present.∎

We now introduce the abstract Schapes Constraint Language (SCL) that containing the main features of ShEx and SHACL and has a quite straightforward translation to both of them described in Appendix A of the long version. Its syntax is very similar to ShEx compact syntax. A shape constraint $\mathsf{Constr}$ is defined by the abstract syntax in Fig. 2 using Extended Backus–Naur Form (EBNF) where the symbol `*' denotes the repetition of an element zero or more times. We next explain the semantics of SCL.

A $\mathsf{Constr}$ is satisfied by a node if this node and its neighbourhood satisfy both the value and the neighbourhood constraints. Each $\mathsf{ValueConstr}$ restricts the value of a node to be one of the options in $\mathsf{ValueConstr}$, where $\mathsf{lit}$ denotes a literal, $\mathsf{nonlit}$ a non-literal, $\mathsf{blank}$ a blank node, $\mathsf{iri}$ any IRI, and $\mathsf{any}$ imposes no value restriction, $\mathsf{XSD}$ defines a required XML Schema Definition type of the value, $\mathsf{Prefix}$ imposes it to be an IRI with the given prefix, and $\mathsf{ListValues}$ enumerates the admissible values. A $\mathsf{NeighbConstr}$ is satisfied by a node if its neighbourhood satisfies all the conjuncts. A $\mathsf{Choice}$ is satisfied when just one of the $\mathsf{TripleConstr}$s composing it is satisfied. A $\mathsf{TripleConstr}$ is to be satisfied by the triples in the neighbourhood of the node having the specified predicate. The number of such triples must fit within the cardinality bounds given by $min$ and $max$, where $min$ is a natural and $max$ is either a natural or the special value '$\mathsf{*}$' for unbounded cardinality. The objects of those triples should be nodes satisfying the specified $\mathsf{ObjectConstr}$, which can be a simple $\mathsf{ValueConstr}$ or another shape. The latter is either a reference to $\mathsf{ShapeLabel}$ or a nested shape with {$\mathsf{NeighbConstr}$}. Note finally that a $\mathsf{NeighbConstr}$ constrains only the triples in the neighbourhood which predicate is mentioned in one of its $\mathsf{TripleConstr}$. All other predicates can be present without restriction on their object value or cardinality.

A schema is a pair $\mathbf{S}=(\mathbf{L},\mathbf{def})$ where $\mathbf{L}\subseteq\mathsf{IRI}$ is a finite set of shape labels and $\mathbf{def}$ associates with every label in $\mathbf{L}$ its definition which is a shape constraint. How an RDF graph should conform to a schema is defined by a validation target. A validation target for schema $\mathbf{S}=(\mathbf{L},\mathbf{def})$ and graph $\mathrm{\mathbf{G}}$ is a pair $\mathbf{T}=(\mathbf{L},\mathbf{tgt})$ where $\mathbf{tgt}$ associates a set of nodes of $\mathrm{\mathbf{G}}$ with every shape label from $\mathbf{L}$. We say that graph $\mathrm{\mathbf{G}}$ satisfies schema $\mathbf{S}$ with target $\mathbf{T}$ if for any shape label $L$ in $\mathbf{L}$, all nodes in $\mathbf{tgt}(L)$ satisfy the constraint $\mathbf{def}(L)$. Most typically, a target is specified using a class from an ontology or a SPARQL query. For instance, one might want to validate all nodes with class City and population greater than 1 million people against a shape <BigCity>. In this paper we consider non-recursive schemas only that do not allow the definition of a shape label to refer to itself neither directly nor transitively.

3 Automatic Construction of Uniform Constraints

Consider a fixed RDF graph $\mathrm{\mathbf{G}}$. In this section we address the problem of automatically constructing a shape constraint to be satisfied by a given sample set of nodes of $\mathrm{\mathbf{G}}$. We define uniform constraints which enjoy the nice property that for any set of sample nodes $N$, there exists a unique most specific uniform constraint satisfied by all the nodes in $N$ (see Sect. 3.1). We also explain how to construct a uniform constraint satisfied by most but not necessarily all sample nodes, thus allowing to take into account noise in the sample data (see Sect. 3.2).

A uniform constraint is defined by the syntax in Fig. 3. We additionally require that the $\mathsf{UTripleConstr}$s that appear in the same $\mathsf{UNeighbConstr}$ are without repeated properties, i.e. their predicates are pairwise distinct. Intuitively, uniform constraints allow to enumerate the predicates to appear in the neighbourhood of a node and to specify for each predicate a constraint for its value and its cardinality among a restricted choice of four possible cardinalities. Remark that uniform constraints belong to the SCL language defined in Sect. 2.

3.1 Most Specific Constraint

Let $F$ be the set of all constraints definable by $\mathsf{ValueConstr}$s and let $\preceq$ be the (partial) ordering relation over $F$ defined by $V_{1}\preceq V_{2}$ if all nodes that satisfy $V_{1}$ also satisfy $V_{2}$. Then for any prefix $pr\textrm{:}$ and for any XSD datatype $X$ it holds

[TABLE]

Moreover, $\preceq$ is defined for prefix constraints by $p\textrm{:}\preceq q\textrm{:}$ if $q$ is a prefix of $p$, and also for XSD datatypes, for instance $\mathsf{int}\preceq\mathsf{decimal}$. Finally, $\preceq$ is defined for $\mathsf{UValue}$ constraints by $[r]\preceq X$ if $r$ is a literal and $X$ is its XSD type; $[r]\preceq p\textrm{:}$ if $r$ is an IRI and $p\textrm{:}$ is its longest prefix, and $[r]\preceq\mathsf{blank}$ if $r$ is a blank node.

Let $K$ be the set of uniform cardinalities, then the subset relation $\subseteq$ on intervals defines an ordering relation over $K$. It is not hard to show that

Lemma 1

$(F,\preceq)$* is an upper semilattice with least upper bound $\bigvee F=\mathsf{any}$. $(K,\subseteq)$ is a complete lattice with least upper bound $\bigvee K=\{0;*\}$ and greatest lower bound $\bigwedge K=\{1;1\}$.*

Therefore, for any set of RDF nodes $R$, there exists a unique most specific uniform value constraint satisfied by all nodes in $R$ that we denote by $\bigvee R$. Similarly, for any set $U$ of natural numbers, there exists a unique most specific uniform cardinality that includes $U$ and we denote it $\bigvee U$.

Definition 1 (Most specific constraint)

Let $N\subseteq\mathrm{\mathbf{N}odes}(G)$ be a sample set of nodes. The most specific constraint associated with $N$ is the uniform constraint $\mathit{msc}(N)$ that contains exactly one $\mathsf{TripleConstr}$ $p~{}V~{}C$ for every predicate $p\in\mathit{props}(N)$ whose value constraint $V$ and cardinality $C$ satisfy:

• •

$V=\bigvee N_{p}$ where $N_{p}$ is the set of $p$-neighbours of $N$;

• •

$C=\bigvee U$ where $U=\{|(n,p,n^{\prime})\in\mathrm{\mathbf{G}}|\mid n\in N\}$.

It easily follows from the definition that

Lemma 2

For any sample $N\subseteq\mathrm{\mathbf{N}odes}(\mathrm{\mathbf{G}})$, it holds that $\mathrm{\mathbf{G}}$ satisfies the schema $(\{S\},[S\mapsto\mathit{msc}(N)])$ with target $(\{S\},[S\mapsto N])$.

The schema defined in Lemma 2 is the most precise schema using uniform constraints that one can construct for matching with the nodes of the sample $N$.

3.2 Largely Accepted Consensus Constraint

The largely accepted consensus constraint allows to account for potential errors in the neighbourhoods of the sample nodes. Imagine that most $p$-neighbours of the nodes in $N$ have an $\mathsf{IRI}$ value, except for a very small number of them whose value is a literal. In this case, $\mathit{msc}(N)$ will consider $\mathsf{any}$ as value constraint for $p$, while it is also possible that the literal $p$-neighbours are errors in the data.

We start by presenting a decision making algorithm used in the definition of the largely accepted consensus constraint. Assume a non-empty set $O$ of options to choose from by a finite set of voters $W$, and let $(O,\preceq)$ be an upper semilattice with least upper bound $o_{\top}$, where $o\preceq o^{\prime}$ indicates that if option $o$ is acceptable for given voter, then $o^{\prime}$ is acceptable for him/her as well. Every individual $n$ votes for its preferred option $\mathit{\pi}(n)\in O$, and let $\textit{acc}_{\mathit{\pi}}(o)$ be the number of voters that consider option $o$ as acceptable. Formally, for any total function $\mathit{\pi}:W\to O$, we define $\textit{acc}_{\mathit{\pi}}(o)=\sum_{o^{\prime}\preceq o}|\{n\mid\mathit{\pi}(n)=o^{\prime}\}|$. It is then easy to see that $\textit{acc}_{\mathit{\pi}}(o_{\top})$ is the total number of votes, i.e. $\textit{acc}_{\mathit{\pi}}(o_{\top})=|W|$.

Let $0.5<t\leq 1$ be a threshold value. We define the consensus option that is acceptable for at least a proportion $t$ of the voters. Let $a=\min\left\{\textit{acc}_{\mathit{\pi}}(o)\mid\frac{\textit{acc}_{\mathit{\pi}}(o)}{|W|}\geq t\right\}$ be the least value in the range of $\textit{acc}_{\mathit{\pi}}$ s.t. $\frac{a}{|W|}$ is above the threshold. Then

[TABLE]

Note that, by definition, $\mathit{consensus}(O,\preceq,W,\mathit{\pi},t)$ is uniquely defined for all semilattices $(O,\preceq)$, sets of voters $W$, vote functions $\mathit{\pi}:W\to O$ and thresholds $t$.

Example 2 (Consensus)

Below on the left, consider the tree (thus also upper semilattice) $(F^{\prime},\preceq)$ with $F^{\prime}\subseteq F$, where $V$ is a child of $V^{\prime}$ whenever $V\preceq V^{\prime}$. Assume a set $W$ of 20 voters and a voting function $\mathit{\pi}:W\to F^{\prime}$. For every option $V$ we give between parentheses $(x;y)$ where $x$ is the number of direct votes (i.e. $x=|\{w\in W|\mathit{\pi}(w)=V\}|$) and $y=\textit{acc}_{\mathit{\pi}}(V)$. Then for different values of the threshold we obtain: $\mathit{consensus}(F^{\prime},\preceq,W,\mathit{\pi},\frac{17}{20})=\mathsf{iri}$, $\mathit{consensus}(F^{\prime},\preceq,W,\mathit{\pi},\frac{18}{20})=\mathsf{nonlit}$ and $\mathit{consensus}(F^{\prime},\preceq,W,\mathit{\pi},\frac{19}{20})=\mathsf{any}$. Below on the right is presented the latitice $K$, again annotated with $(x,y)$ as above for a set $W^{\prime}$ of 20 voters and a voting function $\mathit{\pi}^{\prime}:W^{\prime}\to K$. Then $\mathit{consensus}(K,\subseteq,W^{\prime},\mathit{\pi}^{\prime},\frac{17}{20})=\bigvee\left\{\{0;1\},\{1;*\}\right\}=\{0;*\}$.

\pgfmathresultpt$\mathsf{any}$(0;20)\pgfmathresultpt$\mathsf{nonlit}$(0;18)\pgfmathresultpt$\mathsf{blank}$(1;1)\pgfmathresultpt$\mathsf{iri}$(2;17)\pgfmathresultpthttp://ex.org/:(10;10)\pgfmathresultpt*(5;5)*http://ex.com/:\pgfmathresultpt*(2;2)*$\mathsf{lit}$

          

\pgfmathresultpt{0;}(0;20)\pgfmathresultpt(2;18){1;}\pgfmathresultpt{0;1}(2;18)\pgfmathresultpt{1;1}(16;16)

∎

The largely accepted consensus constraint is defined using a consensus where the voters are (some of) the nodes of the sample $N$. For every $p\in\mathit{props}(N)$, let $N_{p}\subseteq N$ be the set of sample nodes that have at least one $p$-neighbour and assume we are given:

• •

$\mathit{\pi}_{F,p}:N_{p}\to F$ a vote function that gives the preferred value constraint for property $p$ of every sample node that has a $p$-neighbour;

• •

$\mathit{\pi}_{K,p}:N\to K$ a vote function that gives the preferred cardinality for property $p$ of every sample node.

Definition 2 (Largely accepted consensus constraint)

Let $N\subseteq\mathrm{\mathbf{N}odes}(G)$ be a sample set of nodes and $0\leq e<0.5$ be an error rate value. The largely accepted consensus constraint associated with $N$ and assuming error rate $e$ is the uniform constraint $\mathit{lac}_{e}(N)$ that contains exactly one $\mathsf{TripleConstr}$ $p~{}V~{}C$ for every predicate $p\in\mathit{props}(N)$ and it has value constraint $V=\mathit{consensus}(F,\preceq,N_{p},\mathit{\pi}_{F,p},1-e)$, and cardinality $C=\mathit{consensus}(K,\subseteq,N,\mathit{\pi}_{K,p},1-e)$. ∎

The error rate value $e$ is intuitively a bound on the proportion of sample nodes that could have expressed an incorrect222Incorrect w.r.t. a supposedly existing correct uniform constraint that the user aims at. vote due to noise in the data. It is not hard to see that $\mathit{lac}_{0}(N)=\mathit{msc}(N)$.

Finally we define the vote functions $\mathit{\pi}_{F,p}$ and $\mathit{\pi}_{K,p}$. For any node $n\in N$ and any $p\in\mathit{props}(N)$, $\mathit{\pi}_{K,p}(n)$ is defined as follows, where $m$ is the number of $p$-neighbours of $n$:

[TABLE]

As for $\mathit{\pi}_{F,p}$, it is itself defined as a consensus: $\mathit{\pi}_{F,p}(n)=\mathit{consensus}(F,\preceq,M,\textit{best-match},1-e^{\prime})$ where $0\leq e^{\prime}<0.5$ is an error rate that might be different from $e$, the voters $M=\{n^{\prime}\mid(n,p,n^{\prime})\in\mathrm{\mathbf{G}}\}$ are all $p$-neighbours of $n$, and the voting function $\textit{best-match}:M\to F$ is defined by $\textit{best-match}(n^{\prime})=[n^{\prime}]$ (i.e. the singleton $\mathsf{ListValues}$ containing $n^{\prime}$).

Discussion

The $\mathit{msc}$ and $\mathit{lac}$ constraints are implicitly parametrized by $F$ that could be replaced by any upper semilattice. For instance, $F=\{\mathsf{any},\mathsf{blank},\mathsf{lit}\}$ might be sufficient for some use cases. We choose to present this particular value for $F$ because it is useful in practice.

4 Schema patterns

The main limitation of uniform constraints is that they do not allow for nested neighbourhood constraints or references to other shapes. A possible alternative could be to construct a schema up to some fixed nesting depth given as parameter [16, 7] but the same nesting depth is seldom appropriate to all shapes. We propose to use schema patterns instead. They allow for a fine tuning of automatic schema construction by describing a general form for the target schema, but can also be used simply to restrict the namespaces of predicates of interest, or to give a particular role to some values such as the value of rdf:type.

We start by a motivating example based on Wikidata that we use for an informal description of schema patterns in Sect. 4.1. Then in Sec 4.2 we give the syntax of schema patterns.

4.1 Informal Definition

Fig. 4 presents an extract of the Wikidata entry for the city of Auckland333From https://www.wikidata.org/entity/Q37100.ttl retrieved on 2019/03/27.444restricted to the properties rdf:type, P17 and P6.. Entities in Wikidata are encoded by identifiers that start with Q, and properties by identifiers that start with P. The entry in Fig. 4 includes a statements for the property country (P17) with value New Zealand (Q664), and two statements for the property head of government (P6) with values Phil Goff (Q597909) and Len Brown (Q4116955). The most significant value of each property is given as a direct statements (ds) (prefix wdt:), while richer information is presented in a qualified wikibase:Statement in which the claim is accompanied by context or scope information (under which conditions the claim holds), as well es meta information such as provenance. Note that the structures of such statements are similar. They usually contain property statements (ps:), qualifiers (pq:) and provenance information.

In Fig. 5 is presented the $\mathit{msc}(N)$ constraint where the sample $N$ contains the Wikidata entries for Auckland, Monterey, Vienna and Kobe4, which is clearly too poor to represent the actual structure of the data.

A more appropriate schema $\mathbf{S}_{\textrm{city}}$ for cities in Wikidata4 is given in Fig. 6. It accounts for the possible qualifiers of the different properties.

A schema similar to $\mathbf{S}_{\textrm{city}}$ can be automatically constructed using the schema pattern $\mathbf{P}_{\textrm{city}}$ in Fig. 7 as input of the constraint construction algorithm. A schema pattern is similar to a schema but it omits cardinalities, it allows predicate filters (p:, $\mathsf{iri}$, ps:, …) instead of predicates, it replaces value constraints by placeholders $\_\_$ or $[\_\_]$, and it replaces some shape names by variables ($Y$).

The pattern $\mathbf{P}_{\textrm{city}}$ defines one non-variable shape label <City>, therefore it will be used with one set of sample nodes, say $N_{\textrm{city}}$, in order to construct a shapes schema. Denote $P_{\textrm{city}}$ the pattern for <City> and assume that $N_{\textrm{city}}$ contains the single node wd:Q37100 from Fig. 4. We start by matching all properties $q$ in $\mathit{props}(N_{\textrm{city}})$ with the predicate filters that appear in the $P_{\textrm{city}}$, and every such property $q$ is matched with the most specific possible filter. That is p:P17 and p:P6 are matched with p: and we write $\mathit{matched}(\texttt{p:})=\{\texttt{p:P16},\texttt{p:6}\}$, while $\mathit{matched}(\texttt{iri})=\{\texttt{wdt:P17},\texttt{wdt:P6},\texttt{rdf:type}\}$. Now every property in $\mathit{matched}(\texttt{p:})$ yields a triple constraint, and the pattern requires that its object is @Y, a shape reference to a freshly created shape label. We can see that schema $\mathbf{S}_{\textrm{city}}$ in Fig. 6 contains indeed two triple constraints p:P17 @<Y_P17> and p:P6 @<Y_P6>. The cardinalities of these triple constraints are not relevant here as they were obtained by considering a bigger sample set $N_{\textrm{city}}$. In the general case, any property $q\in\mathit{matched}(\texttt{p:})$ defines its corresponding sample $N_{q}=\{n^{\prime}\mid\exists n\in N_{\textrm{city}}.(n,q,n^{\prime})\in\mathrm{\mathbf{G}}\}$ composed of the $q$-neighbours of the nodes in $N_{\textrm{city}}$. This sample is used to determine the cardinality of the triple constraint on the one hand, and as a sample for the definition of shape label <Y_$q$> on the other hand. Similarly, every property in $\mathit{matched}(\mathsf{iri})$ yields a triple constraint which object constraint is a value constraint, as indicated by the placeholder . Now regarding the pattern $P_{Y}$ for $Y$, we already saw that it defines two different shapes as defined by $\mathit{matched}(\texttt{p:})$, and each of these has its associated sample. Remark that $P_{Y}$ does not contain filter $\mathsf{iri}$, therefore some of the properties of the sample nodes might match none of the filters. These properties will not be considered. For instance, property prop:wasDerivedFrom appears in the sample nodes in Fig. 4, but not in the schema $\mathbf{S}_{\textrm{city}}$. We finally explain the triple constraint rdf:type [] in $P_{Y}$. Its predicate holder is not a filter but a property, so it will be matched only by this same property. Its object constraint is the placeholder [__] indicating that we want to construct a $\mathsf{ListValues}$ constraint for the set of values. In the example this is the singleton list that contains wikibase:Statement.

As we saw, schema patterns are particularly fitted for the Wikidata dataset as it strongly relies on reification. But they are also useful for all datasets that use reification [5] or repetitive local structure. Simpler schema patterns can also be used for e.g. restraining the properties of interest.

4.2 Formal Definition

Assume a countable set $\mathcal{V}_{\mathrm{sch}}$ of shape label variables ranging over $X,Y,\ldots$.A constraint pattern (or pattern for short) is defined by the syntax in Fig. 8.

A schema pattern is a triple $\mathbf{P}=(\mathbf{PL},\mathbf{pdef},\mathbf{sample})$ where:

• •

$\mathbf{PL}\subseteq\mathcal{L}\cup\mathcal{V}_{\mathrm{sch}}$ is a finite set of shape labels and shape label variables;

• •

with every label or variable in $\mathbf{PL}$, $\mathbf{pdef}$ associates a shape pattern that uses shape variables from $\mathbf{PL}\cap\mathcal{V}_{\mathrm{sch}}$ only;

• •

$\mathbf{sample}$ associates a set of nodes from $\mathrm{\mathbf{G}}$ with every label in $\mathbf{PL}\cap\mathcal{L}$.

A schema pattern defines a most specific uniform schema (or a largely accepted consensus schema) which definition is given in Appendix Bof the long version due to space limitations.

4.3 Patterns for Ontologies

We show here how a schema pattern can be used to encode some of the information available in an existing ontology that might be relevant for the schema construction. Consider the toy RDFSchema ontology $R$ that defines three classes :Human, :Teacher and :Subject with :Teacher subclass of :Human, and three properties :name, :teaches, :description with axioms

⬇

:name rdfs:domain :Human. :description rdfs:domain :Subject.

:teaches rdfs:domain :Teacher. :teaches rdfs:range :Subject.

We can automatically retrieve the following schema pattern

⬇

<Human> { a [ __ ] ; :name __ }

<Teacher> { a [ __ ] ; :name __ ; :teaches __ }

<Subject> { a [ __ ] ; :description __ }

with $\mathbf{tgt}(\texttt{<Human>})$ defined by the SPARQL query SELECT ?x WHERE ?x a :Human. and similarly for <Teacher> and <Subject> shapes. The shapes schema constructed with this pattern will contain only the properties of the ontology vocabulary attached to their respective domains.

5 Interactive Improvement of the Schema

We are convinced that an optimal shapes schema, i.e. a schema that encodes correctly the requirements for a particular application, cannot be obtained by a fully automatic process. Automatically constructed schemas are necessarily simple: if we allowed the choice operator or repeated properties555that is, allow two triple constraints with same predicate in the same neighbourhood constraint in uniform constraints, then a unique most specific constraint would not exist. Schema patterns (or other forms of parametrization) can bring a partial solution, but they may require user expertise and good knowledge about the data, which is often the reason why we want to build a schema at the first place: a chicken-and-egg situation.

Our solution is to let the user build the schema but within the interactive workflow described hereafter that gives access to automatic schema construction and to useful information about the data, and provides appropriate feedback.

Assume an RDF graph $\mathrm{\mathbf{G}}$ for which we want to build a schema. The interactive workflow maintains a schema $\mathbf{S}=(\mathbf{L},\mathbf{def})$ together with associated validation target $\mathbf{T}=(\mathbf{L},\mathbf{tgt})$ and schema pattern $\mathbf{P}=(\mathbf{PL},\mathbf{pdef},\mathbf{sample})$ with $\mathbf{sample}=\mathbf{tgt}$.A target $\mathbf{tgt}(L)$ is given as a tuple $(Q,N^{+},N^{-})$ of a SPARQL query $Q$ to be executed against $\mathrm{\mathbf{G}}$ and sets of nodes $N^{+}$ and $N^{-}$, to be added, resp. removed from the result of query $Q$. The schema and the targets can be modified using the following editing operations:

• •

modify $\mathbf{def}(L)$ for shape label $L$ using these actions:

• –

add or remove a $\mathsf{Conjunct}$ or $\mathsf{ValueConstraints}$;

• –

group two or more triple constraints into a $\mathsf{Choice}$, or split a choice;

• –

split a $\mathsf{TripleConstr}$ of the form $p~{}V~{}C$ in two $\mathsf{TripleConstr}$s $p~{}V_{1}~{}C_{1}$ and $p~{}V_{2}~{}C_{2}$ having the same predicate, or regroup two such triple constraints;

• –

change the cardinality or the $\mathsf{ObjectConstr}$ of some triple constraints with predicate $p$. In particular, the object constraint can be replaced by the automatically constructed $\{\mathit{msc}(N^{\prime})\}$ or $\{\mathit{lac}(N^{\prime})\}$, where $N^{\prime}$ is the set of $p$-neighbours of $\mathbf{tgt}(L)$;

• •

modify $\mathbf{tgt}(L)$ for shape label $L$. This action is triggered automatically when some triple constraint is changed and becomes of the form $p~{}@L~{}C$ (with shape reference to $L$). Then the set $N^{\prime}$ of all IRI $p$-neighbours of $\mathbf{tgt}(L)$ is added to $\mathbf{tgt}(L^{\prime})$. In the latter case, the $\mathsf{NeighbConstr}$ $M$ is automatically constructed as $\mathit{msc}(N^{\prime})$ or $\mathit{lac}(N^{\prime})$.

• •

add a new shape label $L$ to $\mathbf{L}$ with corresponding target $\mathbf{tgt}(L)=N$ and pattern $\mathbf{pdef}(L)$, which triggers the automatic construction of $\mathit{msc}(N)$ or $\mathit{lac}(N)$ using $\mathbf{pdef}(L)$ as pattern and assigns the result to $\mathbf{def}(L)$.

These operations allow to construct an arbitrarily complex SCL schema starting e.g. from an automatically constructed one.

At any time the user can visualize the result of validating $\mathrm{\mathbf{G}}$ against $\mathbf{S}$ with target $\mathbf{T}$. Additionally, the user has access to the following information:

• •

for any shape label $L$ and triple constraint $p~{}V~{}C$ that occurs in $\mathbf{def}(L)$ or in $\mathbf{pdef}(L)$, and denoting $N=\mathbf{tgt}(L)$:

• –

the number of nodes in $N$ that have $p$ as predicate;

• –

the minimal, maximal and average number of occurrences of predicate $p$ in the neighbourhood of $N$;

• –

the lattice of uniform value constraints $(F,\preceq)$ annotated with $\textit{acc}_{\mathit{\pi}_{F}}(V^{\prime})$ for every $V^{\prime}\in F$ (see Sect. 3.1);

• •

for any pair of predicates $p,p^{\prime}$, the number of nodes from $N$ in which $p$ and $p^{\prime}$ co-occur; this is useful to detect possible $\mathsf{Choice}$s.

5.0.1 Implementation

We have already implemented several components of our method, some of which are illustrated in the video attached to the submission:

• •

the algorithm for automatic construction of the most specific schema with only limited support of schema patterns tailored for Wikidata (video),

• •

simple visualisation of statistics about the data,

• •

enhanced visualisation of validation results that allows to navigate synchronously in the schema and in the graph to understand why a given node is valid or not w.r.t. a given constraint (video).

Currently our implementation is based on ShEx validation. In order to complete the implementation we need to:

(1) complete the implementation of schema patterns;

(2) add support for SHACL validation;

(3) provide a nice visualisation of the information about the data; and

(4) integrate the different components into a single application.

Fig. 9 shows a consolidated view of three of the already implemented components. We consider a sample with four Wikidata entries: Auckland, Monterey, Vienna and Kobe (restricted to a part of their predicates only). The top right panel contains the schema that was automatically constructed using a schema pattern similar to the one presented in Fig. 7. We changed a single cardinality in that schema that produced a validation error. In the bottom panel we see the list of validated nodes on which the error is highlighted. The top left panel shows part of the information provided to the user, i.e. the list of the properties of sample nodes and their frequencies of occurrence.

6 Conclusion and Future Work

We have presented a method that allows to define ShEx or SHACL schemas for existing datasets. The construction of simple (uniform) schemas is fully automatic and is accessible to novice users, while experts can supply their knowledge about the data using schema patterns as an expressive parametrization mechanism, and further edit the resulting schema. A prototype implementation of the method allows to extract and refine ShEx schemas for Wikidata. We are currently extending the implementation in order to integrate the still missing features: full support for schema patterns, translation to SHACL and integrated SHACL validation, a more readable representation of the statistics about the data using GUI widgets. As a future work we are planning to enhance schema patterns in order to capture richer information that comes from ontologies such as cardinalities, and also to allow to describe more complex references between shapes.

Appendix 0.A The Shape Constraint Language, SHACL and ShEx

We explain here how SCL is translated to SHACL and to ShEx, then we explain the small differences in semantics depending on the target language.

Translation to SHACL

SCL's $\mathsf{ValueConstr}$s are represented using sh:nodeKind (for $\mathsf{lit}$, $\mathsf{nonlit}$, $\mathsf{blank}$, $\mathsf{iri}$), using sh:datatype for $\mathsf{XSD}$, using sh:pattern for $\mathsf{Prefix}$, and using sh:in for $\mathsf{ListValues}$. A $\mathsf{NeighbConstr}$ yields a sh:shape that is a conjunction (sh:and) of the encodings of its $\mathsf{Conjunct}$s. The $\mathsf{Choice}$ operator can be represented by sh:or or by sh:xone (which one to use is a parameter of the translation). $\mathsf{TripleConstr}$ is encoded using sh:property with sh:path constraint for its predicate, while its $\mathsf{ObjectConstr}$ is represented by sh:node if it is a shape reference or a nested $\mathsf{NeighbConstr}$, and by the encoding of the $\mathsf{ValueConstr}$ otherwise. Cardinality is represented by sh:minInclusive and sh:maxInclusive as expected. Additionally, if two or more triple constraints in the same neighbourhood constraint have the same predicate, then all these triple constraints are encoded using sh:qualifiedValueShape, and their cardinalities using sh:qualifiedMinCount and sh:qualifiedMaxCount. Finally, SHACL shapes have associated target declarations that are not provided by the SCL schema but by its accompanying validation target. In Fig. 10 we give as example the shapes <Person> and <Date> from Sect. 2 written in SHACL.

Translation to ShEx

The syntax of SCL is based on the compact syntax of ShEx, therefore the translation to ShEx is almost straightforward, except for $\mathsf{Conjunct}$ that becomes EachOf also denoted ; in the compact syntax, and $\mathsf{Choice}$ that becomes OneOf denoted $\mid$. Additionally, cardinality {1;1} is omitted, cardinaity {0;*} becomes , {1;} becomes + and {0;1} becomes ?, while any other cardinality remains as it is. In Fig. 11 we represent the translation to ShEx of shapes <Person> and <Date> from Sect. 2.

Discussion

The semantics of a SCL schema can be different if it is translated to ShEx or to SHACL. The two differences come from the choice operator and from the use of different triple constraints with the same property in the same neighbourhood constraint. This is not an issue for our approach because we make the hypothesis that the user constructs a schema for one of the two languages, not for both, depending on the application she or he has in mind and on her/his production suite. The validation results in the interactive tool are computed using the translation to the chosen target schema language, whichever it is. When the desired abstract SCL schema is ready, it is exported and can be used in production.

Appendix 0.B Formal Definition of Shape Patterns

A schema pattern is defined by the syntax in Fig. 8. We additionally require that (1) no $\mathsf{PredFilter}$ is used more than once in the same constraint pattern, and (2) every shape label variable $Y\in\mathcal{V}_{\mathrm{sch}}$ is used exactly once in the range of $\mathbf{pdef}$.

Denote by $E$ the set of $\mathsf{PredFilter}$s of constraint patterns. Remark that $E$ is a subset of the set $F$ of value constraints defined in Sect. 3, and that $(E,\preceq)$ is an upper semilattice with least upper bound $\bigvee E=\mathsf{iri}$. For any $\mathsf{PNeighbConstr}$ $P$, denote $E(P)$ the set of predicate filters that appear in the direct $\mathsf{PTripleConstr}$s of $P$ (that is, excluding those that appear in some nested $\mathsf{PNeighbConstr}$). Because $E(P)\subseteq E$ and $(E,\preceq)$ is a lattice, it holds that for every property $q$ either no element of $E(P)$ matches $q$ and we write $\mathit{match}(P,q)=\bot$, or there is unique most precise $f$ in $E(P)$ s.t. $[q]\preceq f$ and we write $\mathit{match}(P,q)=f$.

For a property $q$, denote $E(P)_{q\preceq}$ the set of filters in $E(P)$ that match $q$, i.e. $E(P)_{q\preceq}=\{f\in E(P)\mid[q]\preceq f\}$. For instance, if $P$ is $\mathbf{pdef}(Y)$ from Fig. 7 then $ps\textrm{:}\in E(P)$ and $ps\textrm{:}\in E(P)_{\texttt{ps:foo}\preceq}$.

Fix a pattern schema $\mathbf{P}=(\mathbf{PL},\mathbf{pdef},\mathbf{sample})$ for the sequel. Its associated most specific schema with uniform constraints $\mathit{msc}(\mathbf{P})$ is defined below. We point out first that $\mathit{msc}(\mathbf{P})$ is not a uniform schema as it allows nested neighbourhood constraints and shape references, but likewise uniform schemas, its neighbourhood constraints satisfy do not use choice or repeated properties. So let $\mathit{msc}(pat)=(\mathbf{L},\mathbf{def})$, we explain now which are the shape labels in $\mathbf{L}$ and which are their definitions.

For every shape label $L\in\mathbf{PL}$, $L$ is in $\mathbf{L}$ and its its definition $P=\mathbf{pdef}(L)$ together with its sample $N=\mathbf{sample}(L)$ define a $\mathsf{NeighbConstr}$ $S=\mathit{msc}(P,N)$. $S$ is a conjunct of $\mathsf{TripleConstr}$s (without $\mathsf{Choice}$s). Denote by $\textit{filters}(P)$, resp. by $\textit{predicates}(P)$, the set of $\mathsf{Pred}$icates, resp. of $\mathsf{PredFilter}$s, that occur in the $\mathsf{PTripleConstr}$s of $P$. For instance, $\textit{filters}(\mathbf{pdef}(Y))=\{\texttt{ps:},\,\texttt{pq:}\}$ and $\textit{predicates}(\mathbf{pdef}(Y))=\{\texttt{rdf:type}\}$ for the pattern for shape $Y$ in Fig. 7. Then let $\mathit{props}(P,N)$ be the set that contains exactly the properties that apper in $P$ and all properties that appear in the neighbourhood of $N$ and match some of the filters in $P$.

[TABLE]

Finally, for every predicate $q\in\mathit{props}(P,N)$, its associated sample $s(q)$ is the set of $q$-neighbors of $N$.

Each $q\in\mathit{props}(P,N)$ yields a triple constraint $q~{}O~{}C$ which object constraint $O$ and cardinality constraint $C$ are defined as follows. Let $H_{q}$ be the object constranint holder for $q$ by $P$. That is, if $q\in\textit{predicates}(P)$, then $H_{q}$ is the object constraint holder associated with $q$ in $P$. Otherwize, $H_{q}$ is the object constraint holder associated with $\mathit{match}(P,q)$, the most precise predicate filter that matches $q$.

Now, the object constraint $O$ is, depending on the form of $H_{q}$, and denoting $N_{q}$ the set of $q$-neighbours of $N$:

• •

If $H_{q}=\_\_$ (a value constraint holder), then $O=\bigvee N_{q}$.

• •

If $H_{q}=[\_\_]$ (a list of values holder), then $O$ is the list that contains exactly the values in $N_{q}$.

• •

If $H_{q}=\{M\}$ (a nested neighbourhood constraint pattern), then $M=\mathit{msc}(P,N_{q})$.

• •

If $H_{q}$ is of the form $@X$ (shape reference), then we create a fresh shape label <X_q> and $O=@\texttt{<X\_q>}$. We add <X_q> to $\mathbf{L}$ and its definition $\mathbf{def}(\texttt{<X\_q>})=\mathit{msc}(\mathbf{pdef}(X),N_{q})$.

Because the pattern $\mathbf{P}$ is not recursive (requirement (2) above), this procedure terminates and construcs a shapes schema.

We finally point out that schema patterns might suffer of the problem of ``shrinking samples''. Even if the size of $\mathbf{sample}(L)$ for some $L\in\mathbf{PL}$ is sufficient, its nested neighbourhood constraints are constructed using samples of $q$-neighbours of the nodes in $N$ for the appropriate predicate $q$. If predicate $q$ occurs only a few times in the neighbourhood of $N$, then the corresponding sample can be very small. In our implementation we warn the user if such situation occurs, and allow her to add additional nodes to that sample without necessarily augmenting $\mathbf{sample}(L)$ (for efficiency and readability reasons).