Safe-DS: A Domain Specific Language to Make Data Science Safe

TL;DR

Safe-DS is a domain-specific language that enhances the safety and reliability of data science pipelines written in Python by enabling static type checking, range validation, and graphical development tools.

Contribution

It introduces Safe-DS, a DSL that integrates with Python libraries, providing advanced static analysis and a graphical interface to improve data science pipeline safety.

Findings

Detects type errors, range violations, and call order issues

Automatically generates stubs from Python libraries

Provides a synchronized textual and graphical development environment

Abstract

Due to the long runtime of Data Science (DS) pipelines, even small programming mistakes can be very costly, if they are not detected statically. However, even basic static type checking of DS pipelines is difficult because most are written in Python. Static typing is available in Python only via external linters. These require static type annotations for parameters or results of functions, which many DS libraries do not provide. In this paper, we show how the wealth of Python DS libraries can be used in a statically safe way via Safe-DS, a domain specific language (DSL) for DS. Safe-DS catches conventional type errors plus errors related to range restrictions, data manipulation, and call order of functions, going well beyond the abilities of current Python linters. Python libraries are integrated into Safe-DS via a stub language for specifying the interface of its declarations, and an API-Editor that is able to extract type information from the code and documentation of Python libraries, and automatically generate suitable stubs. Moreover, Safe-DS complements textual DS pipelines with a graphical representation that eases safe development by preventing syntax errors. The seamless synchronization of textual and graphic view lets developers always choose the one best suited for their skills and current task. We think that Safe-DS can make DS development easier, faster, and more reliable, significantly reducing development costs.

Full text

In this paper, we show how the wealth of Python DS libraries can be used in a statically safe way via Safe-DS, a domain specific language (DSL) for DS. Safe-DS catches conventional type errors plus errors related to range restrictions, data manipulation, and call order of functions, going well beyond the abilities of current Python linters. Python libraries are integrated into Safe-DS via a stub language for specifying the interface of its declarations, and an API-Editor that is able to extract type information from the code and documentation of Python libraries, and automatically generate suitable stubs.

Moreover, Safe-DS complements textual DS pipelines with a graphical representation that eases safe development by preventing syntax errors. The seamless synchronization of textual and graphic view lets developers always choose the one best suited for their skills and current task.

We think that Safe-DS can make DS development easier, faster, and more reliable, significantly reducing development costs.

††publicationid: pubid:

Accepted at the 45th Int. Conf on Software Engineering, NIER Track

ICSE-NIER’23, May 14–20, 2023, Melbourne, Australia

©2023 IEEE

I Introduction

Various researchers have noted that the APIs of widely-used libraries are often hard to learn and use correctly [1, 2, 3, 4]. Prevention of erroneous use is particularly crucial in Data Science (DS): Programs that crash at run-time for a reason that could have been detected at compile-time are generally a waste of time, resources, and, ultimately, money – and downright prohibitive after running hours or days. Therefore, static safety would be a major benefit for every Data Scientist, novice or master. Unfortunately, most DS pipelines are written in Python, and use Python DS libraries [5]. For Python, static type checking is only available via external linters like mypy111https://github.com/python/mypy, which are, however, inapplicable if the code is not annotated with type hints [6]. Unfortunately, many libraries do not provide type hints, and no Python linter we know of is able to handle advanced type concepts like schema types for tabular datasets, which would help a lot in typical DS applications.

A radically different approach is taken by systems that hide the DS APIs behind a visual pipeline specification language (e.g. RapidMiner Studio [7]), a wizard-like interface (e.g. RapidMiner Go [8]), or a configuration language (e.g. Ludwig [9]). However, wizards and configuration languages can only provide guidance [10] and safety at the expense of limiting what can be expressed. Visual programming can offer full expressiveness, but when a pipeline gets complex, a graph representation quickly gets cluttered, and textual languages excel. Depending on the skills of the user and the use case, different representations are optimal. However, no known system provides visual and textual ways to specify DS pipelines.

In this context, we explore the following research questions:

• •

RQ 1: How can we make the most widely used libraries for DS usable even by programming novices?

• •

RQ 2: Is it possible to design a language that is expressive enough to implement any DS pipeline but simple enough to be learned and used correctly, even by novices?

• •

RQ 3: Which classes of errors are not caught by conventional type systems (especially those of Python linters) but occur frequently in DS development?

• •

RQ 4: Which type concepts could help and should be part of a dedicated DS language?

• •

RQ 5: How can a DS language ensure statically safe use of DS libraries written in dynamic languages?

• •

RQ 6: How can we make integration of existing Python DS libraries easy, even for third parties?

• •

RQ 7: How can we leverage the built-in error prevention of graphical languages without sacrificing the flexibility of textual code?

The solution suggested in this paper is Safe-DS, a statically typed domain specific language (DSL) [11] for specifying DS pipelines. It provides:

A simple mechanism to integrate existing Python DS libraries in a type-safe way (Sec. IV). 2. 2.

A well-designed standard library that includes the best available DS libraries (briefly outlined in Sec. VI). 3. 3.

Support for third-party developers to extend the standard library by integrating their own features (Sec. IV-C). 4. 4.

A seamless bidirectional transition between graphical and textual representation, which are just different views of the same abstraction, the DSL (Sec. V).

II State of the Art

According to the 2021 Kaggle DS survey [5], over 84% of almost 26.000 respondents use Python regularly, making it the most commonly used DS programming language. According to the survey, the most-often used libraries for DS are Matplotlib [12], Seaborn [13], and Plotly [14] for data visualization, Pandas [15] for data processing, and scikit-learn [16, 17], TensorFlow [18], Keras [19], PyTorch [20], and XGBoost [21] for ML222Each group is sorted by descending usage percentages[5]. All are implemented in Python, emphasizing its role in DS development.

Nevertheless, there are noteworthy suggestions for DS pipeline development beyond Python: Ludwig [9] allows users to build a DS pipeline with a YAML configuration file. Users must specify input and output features, and can optionally alter some parts of the pipeline, such as data preprocessing.

External domain specific languages are another solution. The list of DSLs used for DS in 2016 [22] includes textual DSLs like Pig Latin [23] for data processing and visual DSLs like [24] for the representation of probabilistic models. Since then, other tools appeared, e.g. BiDaML [25, 26], which offers five diagrammatic notations for different stages of the development of a DS pipeline. Another popular tool is RapidMiner Studio (formerly YALE) [7], which offers a visual DSL to quickly, and safely build a DS pipeline.

Finally, some tools use wizards to guide users step by step through the development of a DS pipeline, for example RapidMiner Go [8]. To overcome the limited flexibility of a wizard, users can export the final result to RapidMiner Studio.

[27] extends Jupyter Notebook [28] by additional GUI elements like a spreadsheet view to inspect data. When users manipulate the data in these elements, corresponding code is added to the notebook. However, changes to the generated code can in general not be reflected in the GUI, making the transition between graphical and textual view one-way only.

These ideas can be combined with AutoML [29] to automatically derive parts of a DS pipeline from the initial data.

An overview of free tools for data mining can be found in [30] and a more thorough survey of the DS landscape in [31]. Even taking this larger context into account, no existing tool provides a comprehensive solution to the majority of our research questions, let alone all of them.

III A Language for Data Science Pipelines

A DS pipeline is code that performs data loading, data preparation, model training, and hyperparameter optimization to produce a model that is ready for use. It conserves the steps necessary to reproduce the same model from the same data, without the data analysis and model quality assessment steps that developers performed to decide how to build the pipeline.

The main aim of our work is to support pipeline developers of varying skill levels, including absolute programming and DS novices. For this, assembling DS pipelines should be easy to learn and use, but hard to misuse. Any solution must provide a small and consistent API (to ease learning and correct use), which must be easily extensible (to take advantage of the evolution of DS libraries). Extension must be possible and easy, also for third-party library developers.

RQ 1

Given that the main DS libraries are implemented in Python, their use by programming novices for building pipelines is out of question. Even for experienced programmers, Python libraries that lack type annotations for linters are easy to misuse [32]. To make the wealth of DS libraries (Sec. II) usable by novices and prevent misuse by users of all skill levels we propose to reuse these libraries via Safe-DS, a DSL for DS that does the heavy lifting.

RQ 2

This raises the question, which language features are necessary for unrestricted pipeline development and which must be avoided to ease learning and use by novices. Because pipeline development is not about algorithm implementation but mainly a sequence of invocations of already implemented library functions, the set of necessary language features is simple. Fig. 1 shows the start of a Safe-DS pipeline using the popular “Titanic” example.

The keyword pipeline is the entry point into a Safe-DS program. The one in our example is called predictTitanicSurvival. A pipeline contains a sequence of statements, usually assignments333The only other statement, besides assignments, is the expression statement, which evaluates an expression for its side effects, ignoring results.. The right-hand side of each assignment contains the expression to evaluate. An expression can be a literal (''Titanic''), a reference to a variable (titanic), or a call. A call invokes a global function, like loadDataset, or a method, like keepColumns. We refer to methods and global functions as processes. Calls can contain further expressions as arguments.

Processes can take lambdas (anonymous, parameterized sequences of statements) as arguments, to configure parts of their behavior. For example, a general transform function that changes values in one column of a tabular dataset needs to know how to actually change these values. Such higher-order processes are mainly used for data processing functions and rarely needed otherwise.

Notably, in Safe-DS there are no conditionals, loops, or recursive calls. These are encapsulated in the library functions invoked in the pipeline (e.g. hyperparameter search). Decisions and iterations by the developer (reflect on the outcome of the current pipeline, modify it, and restart it) happen outside the pipeline. The lack of loops and conditionals makes the language easy to learn. It also eases thorough static checking (Sec. IV) because data flows linearly through the program.

IV Static Checking

Supporting safety during the development of a DS pipeline is one of our key aims. For this, detecting errors statically, that is before programs are run, is crucial, since the execution of DS pipelines can take hours or even days on large datasets. Achieving static checking, raises the research questions RQ 3 to RQ 6.

IV-A Type Problems

The empirical evaluation of [32] showed that basic typing features of Python linters already catch many common errors. However, we also identified typical problems that go beyond conventional type systems and are particularly frequent or important in a DS context (RQ 3).

Lack of range restrictions

We found that the documentation of a library often expresses range restriction, e.g. ratios must be a numeric value between 0 and 1.

Broken dataset access

When processing tabular datasets by adding, removing or renaming columns, it is easy to forget updating some operations that still access a changed column, because column names are just strings and the type system does not know about their special semantics. Neither does it know how to infer the type of columns.

Broken order

Machine learning processes must often be called in a specific order (e.g. training before prediction). This simple example could be solved at API level by distinguishing the types UntrainedModel and TrainedModel, where only the latter would have a predict method. However, this would exclude integration of existing libraries that mutate an untrained model in place (e.g. scikit-learn), would assume redesign of the library, and would bloat the number of types.

IV-B Type System

Having identified challenging type problems (RQ 3) we now introduce the solutions provided by Safe-DS (RQ 4).

Basic type system

To capture the basic aspects of DS APIs, the Safe-DS type system supports class types, enum types, function types (which include types for parameters and results), union types, subtyping, and genericity.

Refined types

As a way to express range restrictions, Safe-DS additionally includes refined types, which restrict the legal set of values of an existing type via some constraint. For parameters that have a refined type, only expressions that can be evaluated to a constant value by the compiler can be passed. Otherwise, the compiler would not be able to verify the constraint of the refined type.

Schema inference and checking

To address the broken dataset access problem (Sec. IV-A ) Safe-DS checks correctness of schema-sensitive operations, without having to run the entire DS pipeline up to that point: (1) When a tabular dataset is loaded from external sources, e.g. a CSV file, the compiler infers its initial schema by analyzing the data. (2) For any data processing function invoked in the pipeline, the compiler inductively infers the schema of the resulting dataset from the schemata of the input datasets and a description of the function’s schema effects. Our schema effect system [33, 34] captures any combination of column addition, removal, renaming, or type change. In addition, we support two basic schema checks: (1) Ensuring a column with a given name exists. (2) Ensuring a column has a given type. Based on the results of schema inference, the compiler executes such checks statically, thus ensuring early feedback to the user.

Order checking

To solve order issues, we support behavior protocols [35] in interface descriptions. These are regular expressions that describe legal call orders, with processes instead of characters as basic tokens. For instance, we can specify that on any model, the process that trains the model must be called before the one that makes predictions. Using this information, we can ensure statically that the call order in pipelines is correct.

IV-C Integrating External Code

So far, we have not explained, how the Safe-DS compiler can know at all which functions are available, and how it can statically check calls to functions implemented in a language that does not provide a static type system (RQ 5).

Hard-coding a fixed set of processes into the grammar of Safe-DS would clearly preclude extensibility. This could be avoided by analyzing the code of external libraries, collecting contained type hints or inferring the types of arguments and results while the user develops a pipeline. However, many Python DS libraries do not provide type hints and type inference is not generally possible for Python.

Therefore, we designed Safe-DS as a combination of the pipeline language described in Sec. III with an additional stub language. Each stub is an accurate description of the fully typed interface of a particular Python function. Fig. 2 shows the stub for the loadDataset process. The fun keyword indicates that loadDataset is a function. For each parameter (in parentheses) or result (after arrow) we list its name and type, separated by a colon.

Pipelines can now invoke the specified loadDataset process (Fig. 1). The Safe-DS compiler checks type-correctness of invocations, by comparing the types of arguments (inferred) with those specified in the respective stub. Any process call for which there is no matching stub is assumed to be wrong, leading to an error message.

Stubs tell the Safe-DS compiler all it needs to know for type checking. Therefore, the stub language also includes ways to specify refined types, schema effects, schema checks, and behaviour protocols. For example, the process get_column in Fig. 3, which returns the column of the table with the given name, specifies that a column called name must exist in table.

IV-D Semi-Automated Stub Generation and Implementation

With the stub language, implementers of new machine learning models or other useful functions can integrate them into Safe-DS. However, integrating an entire library manually would be tedious and a major deterrent to the adoption of Safe-DS. Thus, we need to explain how this process can be eased (RQ 6).

Our approach adopts and evolves the API-Editor approach of [32]. In a first step, types are inferred for Python code based on type hints in the code [6] or, if type hints are not available, documentation. For the cases that cannot be inferred automatically, an easy to use GUI is provided. It also supports specification of changes to the API (e.g. removal of declarations, renaming, and creation of parameter objects). Some of these suggestions can even be inferred automatically. Once the improved and simplified API is finalized, Safe-DS stubs and their corresponding implementations as Python wrappers around the original library are created automatically.

This way, large libraries can quickly be integrated into Safe-DS, with only limited manual intervention for cases where the documentation is not accurate enough.

V Different Views

So far, we discussed representing Safe-DS pipelines as text (Sec. III). However, this is only suitable for people with some programming skills, which does not apply to everyone involved in the development of DS pipelines. To encourage people without programming skills to try solving DS problems, and understand solutions of experts, we provide different views onto DS pipelines, tailored for specific skillsets (RQ 7).

Fig. 4 shows a possible translation of the pipeline from Fig. 1 into a visual DSL, which still offers full flexibility but improves safety by preventing syntax errors. The surrounding box corresponds to the pipeline, with the name shown in the top right corner. The rounded rectangles inside correspond to the invoked processes, with circles for parameters (left) and results (right). This information is retrieved from the stub of the process (Sec. IV-C).

Arrows represent dataflow between processes, e.g. the arrow from the loadDataset to the keepColumns process says that the result of the first process is input to the second one. The name shown above an arrow corresponds to the name of a variable in the textual DSL. The rightmost arrow is attached to a “blank” circle since this value is never used but we still want to show the name of the variable that stores it, for easier mapping to the textual representation and as a hint to possibly unfinished pipelines. Literal inputs (e.g. strings) are hidden in the graph representation to avoid clutter. They can be made visible by clicking on a process. This approach helps keep the graph small. It is also used in RapidMiner Studio [7].

Existing systems (Sec. II) provide either a code or a graph view of a pipeline, or in case of [27] both but only with a fully defined transition from graph to code view. We think that supporting both views, including seamless transitions in both direction, is crucial, to let each person involved in the development of a DS pipeline use the view best suited for them, and even switch between the views, as needed. It would, for example, facilitate collaboration between domain experts and experienced Data Scientists [36, 37]: A domain expert could prototype an initial pipeline using the visual DSL and then pass it to a Data Scientist, who elaborates and refines it in the textual DSL. Afterwards, the Data Scientist could explain his solution to the domain expert using again the visual DSL, where they could make further changes together.

The switch between different views is only possible if the target view is at least as expressive as the source view. Since we want to be able to seamlessly switch between textual and graphical view back and forth, this implies that both representations must be equivalent. Our approach trivially guarantees this, otherwise hard to achieve, prerequisite. In Safe-DS, text and graph are just two different notations for the same language.

VI Prototype

Language

We implemented a prototype of Safe-DS444https://github.com/Safe-DS/DSL with Xtext555https://www.eclipse.org/Xtext. At the time of writing, it contains the complete textual syntax for the pipeline and stub languages, scoping, automated formatting of Safe-DS code, generation of Python code, and a vast amount of static checks. Static checking currently supports the basic type system without genericity and refined types. It also supports type inference in pipelines. Checking of schema effects is work in progress and behavior protocols are future work. We use Xtext to create a language server [38], which we make available for Visual Studio Code666https://code.visualstudio.com via a small extension777https://marketplace.visualstudio.com/items?itemName=safe-ds.safe-ds.

Metamodel

The core of the language is the metamodel, the abstract representation of the language elements, also known as the abstract syntax of the language. Individual DS pipelines are instances of this metamodel. The static checks are implemented on top of the metamodel. Each view of the DSL (Sec. V) defines a transition to and from the metamodel. First, this allows the static analysis to be defined independently from a concrete view, which only needs to display errors to the user in an appropriate form. Second, we avoid having to define and implement transitions between all current and future views. The metamodel is implemented with EMF888https://www.eclipse.org/modeling/emf.

Fig. 5 shows the relation between the textual and graphical view and the metamodel. It also shows the one-way transition from the metamodel to Python code generated by the compiler.

Standard library

Moreover, we started work on the Safe-DS standard library. So far, it includes basic functionality to work with tabular data (using pandas [15] internally), visualize this data (using seaborn [13]), and train and evaluate ML models (using scikit-learn [16, 17]). An experimental version of the corresponding Python implementation is available on PyPI999https://pypi.org/project/safe-ds. We will discuss the standard library in more detail in a future publication.

VII Discussion

Accuracy of stubs

The static analysis requires that stubs accurately describe the preconditions of the corresponding Python implementation. A mismatch can either cause valid DSL programs to be rejected or invalid DSL programs to be executed, leading to runtime exceptions or undefined behavior. We aim to reduce the chance of such mismatches by automatically creating stubs for Python declarations where possible (Sec. IV-D). Furthermore, users only need to work with the stub language when they want to integrate functionality that is missing from the standard library. For the standard library itself, we pay particular attention to the accuracy of the stubs.

Omission of language concepts

While the language itself contains neither loops nor conditionals, nothing stops a user from implementing higher-order if or while functions in Python and integrating these into Safe-DS with its stub language. We would treat these like any other function, only using the information from the stubs for static analysis or the representation in the graphical view. On the one hand, this means we ignore the semantics of the if and while constructs for static analysis. On the other hand, such constructs are not needed in the first place since we offer domain specific abstractions like a transform function for tabular datasets, which encapsulate any loops or conditionals in the Python implementation and use features of the DSL like the schema checking (Sec. IV-B) for domain specific validation.

VIII Future Plans

Additional checks

We are working on completing the implementation of schema checking and will then add checking of call order using behavior protocols. We also look into whether we can extend [32] to partially infer schema effects and behavior protocols automatically from documentation.

Standard library

In parallel, we are integrating additional features from Pandas [15] for data preparation, seaborn for visualization [13], and scikit-learn [16, 17] for traditional machine learning, using the already implemented stub language (Sec. IV-C) and automated stub generation (Sec. IV-D).

Usability study

When a sufficient part of the standard library is implemented, we will conduct a quantitative and qualitative comparison of Safe-DS and Python for solving real world problems.

Different views

Currently, the available Visual Studio Code extension for Safe-DS only has a textual view. We plan to also implement a graphical view (Sec. V) using GLSP101010https://www.eclipse.org/glsp/.

IX Conclusions

In this paper we focused on the needs of DS applications, rather than DS libraries. We presented Safe-DS (Sec. III), a DSL for the development of DS pipelines. It combines the expressive power of Python DS libraries with safety via compile-time error detection.

Type-safe integration of Python code is based on creation of stubs (Sec. IV-C) that describe the interface of external functions, following largely automated type extraction from a library’s code and documentation (Sec. IV-D). The current implementation of Safe-DS already supports a full conventional static type system, with manifest types in the stubs, and type inference in DS pipelines. Adaptations of non-standard type concepts, such as effect systems (to check schema-conformance of operations on tabular datasets) and behaviour protocols (to check call order) can help deal with these issues statically, too (Sec. IV-B).

To enable DS pipeline development by people with limited programmings skills and foster collaboration of people with different skill levels, our approach supports seamless transition between textual and graphical incarnations of the language (Sec. V). Overall, we hope that Safe-DS will prove valuable to help novices get started with Data Science.

Acknowledgments

This work was partially funded by the Federal Ministry of Education and Research (BMBF), Germany under the Simple-ML project (grant 01IS18054). We are grateful for all former contributions to the Safe-DS GitHub repository, particularly with respect to schema checking and the standard library. We also thank the reviewers for their numerous constructive comments and suggestions.