A Complete Language for Faceted Dataflow Programs

TL;DR

This paper introduces a comprehensive categorical framework for faceted dataflow programs, providing a more expressive diagrammatic language than traditional DAGs, with implications for data transformation tool implementation.

Contribution

It offers a complete categorical axiomatization and a three-dimensional diagrammatic language for dataflow workflows, surpassing existing DAG-based representations.

Findings

Develops a categorical axiomatization of dataflow programs

Introduces a three-dimensional diagrammatic language for workflows

Suggests implementation in data transformation tools

Abstract

We present a complete categorical axiomatization of a wide class of dataflow programs. This gives a three-dimensional diagrammatic language for workflows, more expressive than the directed acyclic graphs generally used for this purpose. This calls for an implementation of these representations in data transformation tools.

Figures

Equations

$$V^{\ast }([A\times \cdots \times B,\dots ,C\times \cdots \times D])$$

$$V^{\ast }([\alpha ])$$

$$V^{\ast }([f\times U])$$

$$V^{\ast }([J_{\_}T])$$

$$V^{\ast }([E_{\_}T])$$

$$f(x_{\_}1)\wedge g(\gamma (x_{\_}2)[1])$$

$$f(x_{\_}1)\wedge \overline{g}(\gamma (x_{\_}2)[1])$$

$$\overline{f}(x_{\_}1)$$

$$(\alpha ):((C,t_{\_}1),\dots ,(C,t_{\_}n))$$

$$\mapsto \perp$$

Full text

Introduction

In the dataflow paradigm, data processing pipelines are built out of modular components which communicate via some channels. This is a natural architecture to build concurrent programs and has been studied in many variants, such as Kahn process networks [kahn1974semantics], Petri nets [petri1966communication, kavi1987isomorphisms], the LUSTRE language [halbwachs1991synchronous] or even UNIX processes and pipes [walker2009composing]. Each of these variants comes with its own requirements on the precise nature of these channels and operations: for instance, sorting a stream requires the module to read the entire stream before writing the first value on its output stream, which violates a requirement called monotonicity in Kahn process networks, but is possible in UNIX. Categorical accounts of these process theories have been developed, for instance for Kahn process networks [stark1991dataflow, hildebrandt2004relational] or Petri nets [pratt1991modeling, meseguer1992semantics].

In this article, we give categorical semantics to programs in Extract-Transform-Load (ETL) software. These three words refer to the three main steps of most projects carried out with this sort of system. Typically, the user extracts data from an existing data source such as a comma-separated values (CSV) file, transforms it to match a desired schema (for instance by normalizing values, removing faulty records, or joining them with other data sources), and loads it into a more structured information system such as a relational or graph database. In other words, ETL tools let users move data from one data model to another. Because the original data source is typically less structured and not as well curated as the target data store, these operations are also refered to as data cleansing or wrangling.

ETL tools typically let users manipulate their data via a collection of operations which can be configured and composed. The way operations can be composed, as well as the format of the data they act on, represent the main design choice for these tools: it will determine what sort of workflow they can represent naturally and efficiently. We will focus here on the tabular data model popularized the OpenRefine software [openrefine], a widely used open source tool popular in the linked open data and data journalism communities.111See http://openrefine.org/, we encourage viewing the videos or trying the software directly, although this article should be readable with no previous knowledge of the tool. We give a self-contained description of the tool in Section 2.

We propose a complete categorical axiomatization for this data model, using two nested monoidal categories. This gives rise to a three-dimensional diagrammatic language for the workflows, generalizing the widespread graph-based representation of dataflow pipelines. The semantics and the complete axiomatization provided makes it possible to use this model to reason about workflow equivalence using intuitive graphical rules.

This has very concrete applications: at the time of writing, OpenRefine has a very limited interface to manipulate workflows, where the various operations used in the transformation are combined in a simple list. Graph-based representations of workflows are already popular in similar tools but are not expressive enough to capture OpenRefine’s model, due to the use of facets, which dynamically change the route followed by data records in the processing pipeline depending on their values. Our approach solves this problem by giving a natural graphical representation which can be understood with no knowledge of category theory, making it amenable to implementation in the tool itself.

1 Categorical semantics of dataflow

Symmetric monoidal categories model an elementary sort of dataflow pipelines, where the flow is acyclic and deterministic. This is well known in the applied category theory community: for instance, [coecke2010quantum] illustrates it by modelling food recipes by morphisms in such categories.

Definition 1**.**

A symmetric monoidal category (SMC) is a category $\mathcal{C}$ equipped with a symmetric bifunctor $\_\otimes\_:\mathcal{C}\times\mathcal{C}\rightarrow\mathcal{C}$. The tensor product is furthermore required to have a unit $I\in\mathcal{C}$ and to be naturally associative.

Informally, objects of $\mathcal{C}$ are stream types and morphisms are dataflow pipelines binding input streams to output streams. Pipelines can be composed sequentially, binding the outputs of the first pipeline to the second, or in parallel, obtaining a pipeline from both inputs to both outputs. The difference between food and data is that discarding the latter is not frowned upon: data streams can be discarded and copied, which makes the category cartesian.

Definition 2**.**

A cartesian category is a symmetric monoidal category $\mathcal{C}$ equipped with a natural family of symmetric comonoids $(\delta_{\_}A:A\to A\otimes A,\bot_{\_}A:A\to I)$ such that $\bot_{\_}I=1_{\_}I$ and $\delta_{\_}{A\otimes B}=(1_{\_}A\otimes s_{\_}{A,B}\otimes 1_{\_}B)\circ(\delta_{\_}A\otimes\delta_{\_}B)$. If these conditions are satisfied one may write the product as $\times$ instead of $\otimes$.

The comultiplication $\delta_{\_}A$ is the copying map and the counit $\bot_{\_}A$ is the discarding map. One can check that this definition of cartesian category is equivalent to the usual one, where the product is defined as the limit of a two-point diagram. The idea behind defining a cartesian category as a symmetric monoidal category with extra structure is to obtain a graphical calculus for cartesian categories. Indeed, morphisms in a SMC can be represented as string diagrams [selinger2010survey]. In Figure 1 we represent the copying and discarding maps as explicit operations.222We draw morphisms with the domain at the top and the codomain at the bottom. The equations they satisfy can then be stated graphically in Figure 2.

String diagrams for cartesian categories are essentially directed acyclic graphs, and this graph-based representation is used in countless software packages, well beyond ETL tools: for instance, Figure 3 shows a compositing workflow in Blender3D333https://www.blender.org/, where the graph-based representation of the image transformation pipeline is manipulated by the user directly.

2 Overview of OpenRefine

Let us now get into more detail about how OpenRefine works. Loading a data source into OpenRefine creates a project, which consists of a simple data table: it is a collection of rows and columns. To each row and column, a value (possibly null) is associated.

The user can then apply operations on this table. Applying an operation will change the state of the table, usually by performing the same transformation for each row in the table. Example of operations include removing a column, reordering columns, normalizing the case of strings in a column or creating a new column whose values are obtained by concatenating the values in other columns. Users can configure these operations with the help of an expression language which lets them derive the values of a new column from the values in existing columns.

Unlike spreadsheet software, such expressions are fully evaluated when stored in the cells that they define: at each stage of the transformation process, the values in the table are static and will not be updated further if the values they were derived from change in the future. For instance, in the sample project of Figure 4, the first operation creates a Full name column by concatenating the Given name and Family name columns. Applying a second operation to capitalize the Family name column does not change the values in the Full name column.

Another difference with spreadsheet software, where it is possible to reference any cell in the expression defining a cell, is that OpenRefine’s expression language only lets the user access values from the same row. For instance, in the same example project of Figure 4, spreadsheet software would make it easy to compute the sum of all donations in a final row. This is not possible in OpenRefine as this would amount to computing the value of a cell from the value of other cells outside of its own row.

In other words, operations in OpenRefine are applied row-wise and are stateless: no state is retained between the processing of rows. It is therefore simple to parallelize these operations, as they amount to a pure map on the list of rows. This is a simplification: in reality, there are violations of these requirements (for instance, OpenRefine offers a sorting operation, and a records mode which introduces a restricted form of non-locality). Due to the limited space we do not review these violations here.

3 Elementary model of OpenRefine workflows

So far, OpenRefine fits neatly in the dataflow paradigm presented in Section 1. One can view each column of a project as a data stream, which can be assigned a type $t\in T$: in our example project, the first two columns are string-valued and the third contains monetary values. These data streams are synchronous: the values they contain are aligned to form rows. An operation $\alpha\in O$ can be seen as reading values from some columns and writing new columns as output. Because of the synchronicity requirement, an operation really is just a function from tuples of input values on the columns it reads to values on the column it writes.

The schema of a table, which is the list of its column types, can be naturally represented by the product of the objects representing its column types. In the example of Figure 4, the initial table is therefore represented by $S\times S\times M$, where $S$ is the type of strings and $M$ of monetary values. Let us call $\alpha:S\times S\rightarrow S$ the first concatenation operation and $\beta:S\rightarrow S$ the second capitalization operation. Figure 4(d) shows a string diagram which models the workflow of Figure 4.

Definition 3**.**

The category $\mathcal{E}$ of table schema and elementary OpenRefine workflows between them is the free cartesian category generated by a set of datatypes $D$ as objects and a set of operations $O$ as morphisms.

This modelling of OpenRefine workflows makes it easy to reason about the information flow in the project. It is possible to rearrange the operations using the axioms of a cartesian category to show that two workflows produce the same results. We could add some generating equations between composites of the generating operations, such as operations which commute even when executed on the same column for instance.

Without loss of generality, we can assume that the generating operations all have a single generating datatype as codomain, as the cartesian structure makes it possible to represent generic operations as composites of their projections. Under these conditions, morphisms of $\mathcal{E}$ can be rewritten to a normal form, illustrated in Figure 5.

Lemma 1**.**

Any morphism $m\in\mathcal{E}$ can be written as a vertical composite of three layers: the first one only contains copying and discarding morphisms, the second only symmetries and the third only generating operations (identities are allowed at each level).

All three slices in the decomposition above can be further normalized: for instance, the cartesian slice can be expressed in left-associative form, the exchange slice is determined by the permutation it represents and the operation slice can be expressed in right normal form [delpeuch2018normalization]. This gives a simple way to decide the equality of diagrams in $\mathcal{E}$. Of course, deciding equality in a free cartesian category just amounts to comparing tuples of terms in universal algebra. We are only formulating it as a graphical rewriting procedure to lay down the methodoly for the next section.

4 Model of OpenRefine workflows with facets

One key functionality of OpenRefine that we have ignored so far is its facets. A facet on a column gives a summary of the value distribution in this column. For instance, a facet on a column containing strings will display the distinct strings occurring in the column and their number of occurences. A numerical facet will display a histogram, a scatterplot facet will display points in the plane, and so on.

Beyond the use of facets to analyze distributions of values, it is also possible to select particular values in the facet, which selects the rows where these values are found. It is then possible to run operations on these filtered rows only. So far our operations ran on all rows indiscriminately, so we need to extend our model to represent operations applied to a filtered set of rows.

We assume from now on a set $F$ of filters in addition to our set of operations $O$. Each filter $f\in F$ is associated with an object $T_{\_}f\in\mathcal{E}$, the type of data that it filters on. Each filter can be thought of as a boolean expression that can be evaluated for each value $v\in T_{\_}f$, determining if the value is included or excluded by the filter. The type $T_{\_}f$ is not required to be atomic: for instance, in the case of a scatterplot filter, two numerical columns are read.

Definition 4**.**

Let $\mathcal{F}$ be the free co-cartesian category generated as follows. We denote by $[A_{\_}1,\dots,A_{\_}n]$ the product of objects $A_{\_}1,\dots,A_{\_}n$ in $\mathcal{F}$ to distinguish it from the product $\times$ in $\mathcal{E}$. For each object $T\in\mathcal{E}$, $[T]\in\mathcal{F}$ is a generator. Morphism generators are:

1. (i)

For each morphism $\alpha\in\mathcal{E}(T,U)$, there is a generator $[\alpha]:[T]\rightarrow[U]$. 2. (ii)

For each filter $f$ and object $U\in\mathcal{E}$, there is a generator $[f\times U]:[T_{\_}f\times U]\rightarrow[T_{\_}f\times U,T_{\_}f\times U]$.

For each object $T\in\mathcal{E}$, we call $J_{\_}T:[T,T]\rightarrow[T]$ and $E_{\_}T:[]\rightarrow[T]$ the comultiplication and counit provided by the co-cartesian structure.

The axioms satisfied by these generators are stated graphically in Figure 7, with the notations introduced in Figure 6. In addition to these axioms, we require that $[g]\circ[f]=[g\circ f]$ (which is tautological graphically). In other words, $\mathcal{E}$ embeds into $\mathcal{F}$ functorially (but that functor is not monoidal).

The definition above can be interpreted intuitively as follows. An object in $\mathcal{E}$ represents the schema of a table (the list of types of its columns). An object of $\mathcal{F}$ is a list of objects of $\mathcal{E}$, so it represents a list of table schemata. As will be made clear by the semantics defined in the next section, a morphism in $\mathcal{F}:[U,V]\mapsto[W,Z]$ should be thought of as a function mapping disjoint tables of respective schemata $U$ and $V$ to disjoint tables of respective schemata $W$ or $Z$, and row-wise so: depending on its values, a row can end up in either of the output tables. This makes it therefore possible to represent filters as morphisms triaging rows to disjoint tables. A filter $[f\times U]$ operates on tables of schema $T_{\_}f\times U$, and only reads values from the first component to determine whether to send the row to the first or second output table. This treatment of a boolean predicate $A\rightarrow 2$ as a morphism $A\rightarrow A+A$ is similar to that of effectus theory [cho2015introduction]. The comultiplication $J_{\_}T$ is a union, merges two tables of identical schemata together.444In this model, row order does not matter in this model: tables are sets of rows. The counit $E_{\_}T$ is the empty table.

Given the two nested list structures in objects of $\mathcal{F}$, it is natural to represent them as two-dimensional objects, and morphisms of $\mathcal{F}$ become three-dimensional objects, as shown in Figure 6. Figure 7 states the relations satisfied by these generators using this convention.

OpenRefine workflows with filters can be represented by morphisms $\mathcal{F}$. For the converse, we first show that morphisms of $\mathcal{F}$ can be represented in normal form thanks to the following decomposition.

Lemma 2.

Let $m\in\mathcal{F}([A],[B])$ be a morphism with one input sheet and one output sheet. There exists a decomposition $m=z\circ y\circ x\circ[w]$ such that $w\in\mathcal{E}$, $x$ only contains filters, $y$ only contains discarding morphisms, and $z$ only contains unions.

This decomposition can be used to show that all such morphisms arise from OpenRefine workflows, despite the fact that some generators cannot be interpreted as such individually. As stated, this lemma does not provide normal forms yet, as the order of filters in $x$ is not determined. We will see in the proof of Theorem 1 how this can be addressed.

5 Semantics and completeness

We can give set-valued semantics to $\mathcal{E}$ and $\mathcal{F}$ and obtain completeness theorems for our axiomatization of OpenRefine workflows.

Definition 5**.**

A valuation $V$ is given by:

1. (i)

a set $V(T)$ for each basic datatype $T\in\mathcal{E}$; 2. (ii)

a function $V(\alpha):V(A)\rightarrow V(B)$ for each generator $\alpha\in\mathcal{E}(A,B)$, where $V(A)$ is the cartesian product of the valuations of the basic types in $A$; 3. (iii)

a subset $V(f)\subset V(T_{\_}f)$ for each filter $f$.

Any valuation $V$ defines a functor $V^{*}:\mathcal{F}\rightarrow\mathbf{Set}$ as follows:

[TABLE]

Using the decomposition of Lemma 2, we can then show the completeness of our axiomatization for these semantics:

Theorem 1.

Let $d,d^{\prime}\in\mathcal{F}([A],[B])$ be diagrams. Then $d=d^{\prime}$ by the axioms of $\mathcal{F}$ if and only if $V^{*}(d)=V^{*}(d^{\prime})$ for any valuation $V$.

The proof of this theorem is given in appendix. Broadly speaking, it goes by building a valuation where values are syntactic terms, such that a value encodes its entire own history through the processing pipeline. These syntactic values are associated with contexts which record the validity of filter expressions. The decomposition of Lemma 2 is then used to compute normal forms for diagrams, which can be related to the evaluation of the diagram with the syntactic valuation. These normal forms can be computed using a simple diagramatic rewriting strategy, so this also solves the word problem for this signature.

We conjecture that this result generalizes to arbitrary morphisms in $\mathcal{F}$, with multiple input and output tables. However, all OpenRefine workflows have one input and one output table, so the theorem already covers these.

6 Conclusion

We have presented a complete axiomatization of the core data model of OpenRefine. This gives a diagrammatic representation for workflows and an algorithm to determine if two workflows are equivalent up to these axioms.

As future work, this visualization suggested by the categorical model could be implemented in the tool itself. This would make workflows easier to inspect, share and re-arrange. This representation could also be the basis of a more profound overhaul of the implementation of the data model, which would make workflow execution more scalable. The axiomatization could also be extended to account for algebraic equations involving the operations, although it seems hard to preserve completeness and decidability for non-trivial equational theories. Finally, the model could be extended to account for a larger class of operations, for instance order-dependent ones such as sorting, or operations which are not applied row-wise (using OpenRefine’s record mode).

7 Acknowledgements

We thank David Reutter, Jamie Vicary and the anonymous reviewers for their helpful feedback on the project. The author is supported by an EPSRC Studentship.

References for the Appendix