checkmate: Fast Argument Checks for Defensive R Programming

TL;DR

The paper introduces 'checkmate', a C-based R package that enables fast, reliable runtime assertions to improve debugging, correctness, and testing of R code, especially in statistical analysis workflows.

Contribution

It presents a comprehensive set of assertion functions in C for R, enhancing debugging and testing efficiency compared to previous approaches.

Findings

Provides high-performance assertion functions in C for R.

Extends 'testthat' with additional expectation functions.

Enables assertions on arbitrary R objects in compiled code.

Abstract

Dynamically typed programming languages like R allow programmers to write generic, flexible and concise code and to interact with the language using an interactive Read-eval-print-loop (REPL). However, this flexibility has its price: As the R interpreter has no information about the expected variable type, many base functions automatically convert the input instead of raising an exception. Unfortunately, this frequently leads to runtime errors deeper down the call stack which obfuscates the original problem and renders debugging challenging. Even worse, unwanted conversions can remain undetected and skew or invalidate the results of a statistical analysis. As a resort, assertions can be employed to detect unexpected input during runtime and to signal understandable and traceable errors. The package "checkmate" provides a plethora of functions to check the type and related properties of the most frequently used R objects and variable types. The package is mostly written in C to avoid any unnecessary performance overhead. Thus, the programmer can conveniently write concise, well-tested assertions which outperforms custom R code for many applications. Furthermore, checkmate simplifies writing unit tests using the framework "testthat" by extending it with plenty of additional expectation functions, and registered C routines are available for package developers to perform assertions on arbitrary SEXPs (internal data structure for R objects implemented as struct in C) in compiled code.

Tables

Table 1. Table 1: Input and output for some simple mathematical functions from the base package ( R -3.3.1). Outputs marked with “(w)” have issued a warning message.

	Return value of
Input	mean(x)	median(x)	sin(x)	min(x)
numeric(0)	NaN	NA	numeric(0)	Inf (w)
character(0)	NA_real_ (w)	NA_character_	[exception]	NA_character_ (w)
NA	NA_real_	NA	NA_real_	NA_integer_
NA_character_	NA_real_ (w)	NA_character_	[exception]	NA_character_
NaN	NaN	NA	NaN	NaN
NULL	NA (w)	NULL (w)	[exception]	Inf (w)

Full text

1 Defensive Programming in R

Most dynamic languages utilize a weak type system where the type of variable must not be declared, and R is no exception in this regard. On the one hand, a weak type system generally reduces the code base and encourages rapid prototyping of functions. On the other hand, in comparison to strongly typed languages like C/C++, errors in the program flow are much harder to detect. Without the type information, the R interpreter just relies on the called functions to handle their input in a meaningful way. Unfortunately, many of R’s base functions are implemented with the REPL in mind. Thus, instead of raising an exception, many functions silently try to auto-convert the input. E.g., instead of assuming that the input NULL does not make sense for the function mean(), the value NA of type numeric is returned and additionally a warning message is signaled. While this behaviour is acceptable for interactive REPL usage where the user can directly react to the warning, it is highly unfavorable in packages or non-interactively executed scripts. As the generated missing value is passed to other functions deeper down the call stack, it will eventually raise an error. However, the error will be reported in a different context and associated with different functions and variable names. The link to origin of the problem is missing and debugging becomes much more challenging. Furthermore, the investigation of the call stack with tools like traceback() or browser() can result in an overwhelming number of steps and functions. As the auto-conversions cascade nearly unpredictably (as illustrated in Table 1), this may lead to undetected errors and thus to misinterpretation of the reported results.

The described problems lead to a concept called “defensive programming” where the programmer is responsible for manually checking function arguments. Reacting to unexpected input as soon as possible by signaling errors instantaneously with a helpful error message is the key aspect of this programming paradigm. A similar concept is called “design by contract” which demands the definition of formal, precise and verifiable input and in return guarantees a sane program flow if all preconditions hold. The package checkmate assists the programmer in writing such assertions in a concise way for the most important R objects.

2 Related work

Many packages contain custom code to perform argument checks. These either rely on (a) the base function stopifnot() or (b) hand-written cascades of if-else blocks containing calls to stop(). Option (a) can be considered a quick hack because the raised error messages lack helpful details or instructions for the user. Option (b) is the natural way of doing argument checks in R but quickly becomes tedious. For this reason many packages have their own functions included, but there are also some packages on CRAN whose sole purpose are argument checks.

The package assertthat (Wickham, 2013) provides the “drop-in replacement” assert_that() for R’s stopifnot() while generating more informative help messages. This is achieved by evaluating the expression passed to the function assert_that() in an environment where functions and operators from the base package (e.g.  as.numeric() or ‘==‘) are overloaded by more verbose counterparts. E.g., to check a variable to be suitable to pass to the log() function, one would require a numeric vector with all positive elements and no missing values:

assert_that(is.numeric(x), length(x) > 0, all(!is.na(x)), all(x >= 0))

Furthermore, assertthat offers some additional convenience functions like is.flag() to check for single logical values or has_name() to check for presence of specific names. These functions also prove useful if used with see_if() instead of assert_that() which turns the passed expression into a predicate function returning a logical value.

The package assertive (Cotton, 2016) is another popular package for argument checks. Its functionality is split over 15 packages containing over 400 functions, each specialized for a specific class of assertions: For instance, assertive.numbers specialises on checks of numbers and asserive.sets offers functions to work with sets. The functions are grouped into functions starting with is_ for predicate functions and functions starting with assert_ to perform stopifnot()-equivalent operations. The author provides a “checklist of checks” as package vignette to assist the user in picking the right functions for common situations like checks for numeric vectors or for working with files. Picking up the log() example again, the input check with assertive translates to:

assert_is_numeric(x) assert_is_non_empty(x) assert_all_are_not_na(x) assert_all_are_greater_than_or_equal_to(x, 0)

Moreover, the package assertr (Fischetti, 2016) focuses on assertions for magrittr (Bache and Wickham, 2014) pipelines and data frame operations in dplyr (Wickham and Francois, 2016), but is not intended for generic runtime assertions.

3 The checkmate Package

3.1 Design goals

The package has been implemented with the following goals in mind:

Runtime

To minimize any concern about the extra computation time required for assertions, most functions directly jump into compiled code to perform the assertions directly on the SEXPs. The functions also have been extensively optimized to first perform inexpensive checks in order to be able to skip the expensive ones.

Memory

In many domains the user input can be rather large, e.g. long vectors and high dimensional matrices are common in text mining and bioinformatics. Basic checks, e.g. for missingness, are already quite time consuming, but if intermediate objects of the same dimension have to be created, runtimes easily get out of hand. For example, any(x < 0) with x being a large numeric matrix internally first allocates a logical matrix tmp with the same dimensions as x. The matrix tmp is then passed in a second step to any() which aggregates the logical matrix to a single logical value and tmp is marked to be garbage collected. Besides a possible shortage of available memory, which may cause the machine to swap or the R interpreter to terminate, runtime is wasted with unnecessary memory management. checkmate solves this problem by looping directly over the elements and thereby avoiding any intermediate objects.

Code completion

The package aims to provide a single function for all frequently used R objects and their respective characteristics and attributes. For example, the function assertNumeric() provides arguments to check for length, missingness and lower/upper bound. After typing the function name, the code completion of editors which speak R can suggest additional checks for the respective variable type. This context-sensitive assistance often helps writing more concise assertions.

3.2 Naming scheme

The core functions of the package follow a specific naming scheme: The first part (prefix) of a function name determines the action to perform w.r.t. the outcome of the respective check while the second part of a function name (suffix) determines the base type of the object to check. The first argument of all functions is always the object x to check and further arguments specify additional restrictions on x.

3.2.1 Prefixes

There are currently four families of functions, grouped by their prefix, implemented in checkmate:

assert*

Functions prefixed with “assert” throw an exception if the corresponding check fails and the checked object is returned invisibly on success. This family of functions is suitable for many different tasks. Besides argument checks of user input, this family of functions can also be used as a drop-in replacement for stopifnot() in unit tests using the internal test mechanism of R as described in Writing R Extensions (R Core Team, 2016), subsection 1.1.5. Furthermore, as the object to check is returned invisibly, the functions can also be used inside magrittr pipelines.

test*

Functions prefixed with “test” are predicate functions which return TRUE if the respective check is successful and FALSE otherwise. This family of functions is best utilized if different checks must be combined in a non-trivial manner or custom error messages are required.

expect*

Functions prefixed with “expect” are intended to be used together with testthat: the check is translated to an expectation which is then forwarded to the active testthat reporter. This way, checkmate extends the facilities of testthat with dozens of powerful helper functions to write efficient and comprehensive unit tests. Note that testthat is an optional dependency and the expect-functions only work if testthat is installed. Thus, to use checkmate as an testthat extension, checkmate must be listed in Suggests or Imports of a package.

check*

Functions prefixed with “check” return the error message as a string if the respective check fails, and TRUE otherwise. Functions with this prefix are the workhorses called by the “asssert”, “test” and “expect” families of functions and prove especially useful to implement custom assertions. They can also be used to collect error messages in order to generate reports of multiple check violations at once.

The prefix and the suffix can be combined in both “camelBack” and “underscore_case” fashion. In other words, checkmate offers all functions with the “assert”, “test” and “check” prefix in both programming style flavors: assert_numeric() is a synonym for assertNumeric() the same way testDataFrame() can be used instead of test_data_frame(). By supporting the two most predominant coding styles for R, most programmers can stick to their favorite style while implementing runtime assertions in their packages.

3.2.2 Suffixes

While the prefix determines the action to perform on a successful or failed check, the second part of each function name defines the base type of the first argument x, e.g. integer, character or matrix. Additional function arguments restrict the object to fulfill further properties or attributes.

Atomics and Vectors

The most important built-in atomics are supported via the suffixes *Logical, *Numeric, *Integer, *Complex, *Character, *Factor, and *List (strictly speaking, “numeric” is not an atomic type but a naming convention for objects of type integer or double). Although most operations that work on real values also are applicable to natural numbers, the contrary is often not true. Therefore numeric values frequently need to be converted to integer, and *Integerish ensures a conversion without surprises by checking double values to be “nearby” an integer w.r.t. a machine-dependent tolerance. Furthermore, the object can be checked to be a vector, an atomic or an atomic vector (a vector, but not NULL).

All functions can optionally test for missing values (any or all missing), length (exact, minimum and maximum length) as well as names being

(a) not present,

(b) present and not NA/empty,

(c) present, not NA/empty and unique, or

(d) present, not NA/empty, unique and additionally complying to R’s variable naming scheme.

There are more type-specific checks, e.g. bound checks for numerics or regular expression matching for characters. These are documented in full detail in the manual.

Scalars

Atomics of length one are called scalars. Although R does not differentiate between scalars and vectors internally, scalars deserve particular attention in assertions as arguably most function arguments are expected to be scalar. Although scalars can also be checked with the functions that work on atomic vectors and additionally restricting to length 1 via argument len, checkmate provides some useful abbreviations: *Flag for logical scalars, *Int for an integerish value, *Count for a non-negative integerish values, *Number for numeric scalars and *String for scalar character vectors. Missing values are prohibited for all scalar values by default as scalars are usually not meant to hold data where missingness occurs naturally (but can be allowed explicitly via argument na.ok). Again, additional type-specific checks are available which are described in the manual.

Compound types

The most important compound types are matrices/arrays (vectors of type logical, numeric or character with attribute dim) and data frames (lists with attribute row.names and class data.frame storing atomic vectors of same length). The package also includes checks for the popular data.frame alternatives data.table (Dowle et al., 2014) and tibble (Wickham et al., 2016). Some checkable characteristics conclude the internal type(s), missingness, dimensions or dimension names.

Miscellaneous

On top of the already described checks, there are functions to work with sets (*Subset, *Choice and *SetEqual), environments (*Environment) and objects of class “Date” (*Date). The *Function family checks R functions and its arguments and *OS allows to check if R is running on a specific operating system. The functions *File and *Directory test for existence and access rights of files and directories, respectively. The function *PathForOutput allows to check whether a directory can be used to store files in it. Furthermore, checkmate provides functions to check the class or names of arbitrary R objects with *Class and *Names.

Custom checks

Extensions are possible by writing a check* function which returns TRUE on success and an informative error message otherwise. The exported functionals makeAssertionFunction(), makeTestFunction() and makeExpectationFunction() can wrap this custom check function to create the required counterparts in such a way that they seamlessly fit into the package. The vignette demonstrates this with a check function for square matrices.

3.3 DSL for argument checks

Most basic checks can alternatively be performed using an implemented Domain Specific Language (DSL) via the functions qassert(), qtest() or qexpect(). All three functions have two arguments: The arbitrary object x to check and a “rule” which determines the checks to perform provided as a single string. Each rules consist of up to three parts:

The first character determines the expected class of x, e.g. “n” for numeric, “b” for boolean, “f” for a factor or “s” for a string (more can be looked up in the manual). By using a lowercase letter, missing values are permitted while an uppercase letter disallows missingness. 2. 2.

The second part is the length definition. Supported are “?” for length 0 or length 1, “+” for length $\geq 1$ as well as arbitrary length specifications like “1”/“==1” for exact length 1 or “<10” for length $<10$. 3. 3.

The third part triggers a range check, if applicable, in interval notation (e.g., “$[0,1)$” for values $0\leq x<1$). If the boundary value on an open side of the interval is missing, all values of x will be checked for being $>-\infty$ or $<\infty$, respectively.

Although this syntax requires some time to familiarize with, it allows to write extensive argument checks with very few keystrokes. For example, the previous check for the input of log() translates to the rule "N+[0,]". As the function signature is really simplistic, it is perfectly suited to be used from compiled code written in C/C++ to check arbitrary SEXPs. For this reason checkmate provides header files which foreign packages can link against. Instructions can be found in the package vignette.

4 Benchmarks

This small benchmark study picks up the $\log()$ example once again: testing a vector to be numeric with only positive, non-missing values.

4.1 Implementations

Now we compare checkmate’s assertNumeric() and qassert() (as briefly described in the previous Section DSL for argument checks) with counterparts written with R’s stopifnot(), assertthat’s assert_that() and a series of assertive’s assert_*() functions:

checkmate <- function(x) { assertNumeric(x, any.missing = FALSE, lower = 0) }

qcheckmate <- function(x) { qassert(x, "N[0,]") }

R <- function(x) { stopifnot(is.numeric(x), all(!is.na(x)), all(x >= 0)) }

assertthat <- function(x) { assert_that(is.numeric(x), all(!is.na(x)), all(x >= 0)) }

assertive <- function(x) { assert_is_numeric(x) assert_all_are_not_na(x) assert_all_are_greater_than_or_equal_to(x, 0) }

To allow measurement of failed assertions, the above functions are wrapped into a try(). The source code for this benchmark study is provided in the the supplementary.

4.2 Setup

The benchmark was performed on an Intel i5-6600 with $16\text{\,}\mathrm{GB}$ running R-3.3.1 on a 64bit Arch Linux installation. The package versions are 1.8.2 for checkmate, 0.1 for assertthat and 0.3-4 for assertive. R, the linked OpenBLAS and all packages have been compiled with the GNU Compiler Collection (GCC) in version 6.2.1 and tuned with march=native on optimization level -O2. To compare runtime differences, microbenchmark (Mersmann, 2015) is setup to do 100 replications. The wrappers have also been compared to their byte-compiled version (using compiler::cmpfun) with no notable difference in performance, thus the later presented results are extracted from the uncompiled versions of these wrappers.

4.3 Results

The benchmark is performed on four different inputs and the resulting timings are presented in Figure 1.

Note that the runtimes on the $x$-axis are on $\log_{10}$-scale and use different units of measurement.

top left

Input x is a scalar character value, i.e. of wrong type. This benchmark serves as a measurement of overhead: the first performed and cheapest assertion on the type of x directly fails. In fact, all assertion frameworks only require microseconds to terminate. R directly jumps into compiled code via a Primitive and therefore has the least overhead. checkmate on the other hand has to jump into the compiled code via the .Call interface which is comparably slower. The implementation in assertthat is faster than checkmate (as it also primarily calls primitives) but slightly slower than stopifnot(). The implementation in assertive is the slowest. However, in case of assertions (in comparison to tests returning logicals), the runtimes for a successful check are arguably more important than for a failed check because the latter raises an exception which usually is a rare event in the program flow and thus is not time-critical. Therefore, the next benchmark might be more relevant for many applications.

top right

Input x is a scalar numeric value. The implementations now additionally check for missingness and negative values and do not raise an exception. qassert() is the fastest implementation, followed by assertNumeric(). Although qassert() and assertNumeric() basically call the same code internally, qassert() has less overhead due to its minimalistic interface. R’s stopifnot() is a tad slower comparing the median runtimes but still faster than assertthat (5x slowdown in comparison to qassert()). assertive is >60x slower than qassert().

bottom left

Input x is now a long vector with $10^{6}$ numeric elements. checkmate has the fastest versions with a speedup of approximately 3.5x compared to R’s stopifnot() and assert_that(). In comparison to its alternatives, checkmate avoids intermediate objects as described in Design goals: Instead of allocating a logical(1e6) vector first to aggregate it in a second step, checkmate directly operates on the numeric input. That is also the reason why stopifnot() and assertthat() have high variance in their runtimes: The garbage collector occasionally gets triggered to free memory which requires a substantial amount of time.

assertive is orders of magnitude slower for this input (>1200x) because it follows a completely different philosophy: Instead of focusing on speed, assertive gathers detailed information while performing the assertion. This yields report-like error messages (e.g., the index and reason why an assertion failed, for each element of the vector) but is comparably slow.

bottom right

Input x is again a large vector, but the first element is a missing value. Here, all implementations first successfully check the type of x and then throw an error about the missing value. Again, checkmate avoids allocating intermediate objects which in this case yields an even bigger speedup: While the other packages first check $10^{6}$ elements for missingness to create a logical(1e6) vector which is then passed to any(), checkmate directly stops after analyzing the first element of x. This obvious optimization yields a speedup of 20x in comparison to R and assertthat and a 6000x speedup in comparison to assertive.

Summed up, checkmate is the fastest option to perform expensive checks and only causes a small decrease in performance for trivial, inexpensive checks which fail quickly (top left). Although the runtime differences seem insignificant for small input (top right), the saved microseconds can easily sum up to seconds or hours if the respective assertion is located in a hot spot of the program and therefore is called millions of times. For large input, the runtime differences are often notable without benchmarks, and even become much more important as data grows bigger.

5 Conclusion

Runtime assertions are a necessity in R to ensure a sane program flow, but R itself offers very limited capabilities to perform these kind of checks. checkmate allows programmers and package developers to write assertions in a concise way without unnecessarily sacrificing runtime performance nor increasing the memory footprint. Compared to the presented alternatives, assertions with checkmate are faster, tailored for bigger data and (with the help of code completion) more convenient to write. They generate helpful error messages, are extensively tested for correctness and suitable for large and extensive software projects (mlr (Bischl et al., 2016) and BatchJobs (Bischl et al., 2015) already make heavy use of checkmate). Furthermore, checkmate offers capabilities to do assertions on SEXPs in compiled code via a domain specific language and extends the popular unit testing framework testthat with many helpful expectation functions.

6 Acknowledgments

Part of the work on this paper has been supported by Deutsche Forschungsgemeinschaft (DFG) within the Collaborative Research Center SFB 876 “Providing Information by Resource-Constrained Analysis”, project A3 (http://sfb876.tu-dortmund.de).