archivist: An R Package for Managing, Recording and Restoring Data Analysis Results

TL;DR

The paper introduces the archivist R package, which enhances data analysis management by archiving, sharing, and restoring R objects along with their metadata, relations, and context to improve reproducibility and collaboration.

Contribution

It presents new functionalities for managing R objects and their metadata, extending reproducible research with object retrieval, validation, and pedigree tracking capabilities.

Findings

Enables archiving and sharing R objects with metadata and relations.

Supports retrieval and validation of objects for reproducibility.

Allows interactive exploration and pedigree tracking of R objects.

Abstract

Everything that exists in R is an object [Chambers2016]. This article examines what would be possible if we kept copies of all R objects that have ever been created. Not only objects but also their properties, meta-data, relations with other objects and information about context in which they were created. We introduce archivist, an R package designed to improve the management of results of data analysis. Key functionalities of this package include: (i) management of local and remote repositories which contain R objects and their meta-data (objects' properties and relations between them); (ii) archiving R objects to repositories; (iii) sharing and retrieving objects (and it's pedigree) by their unique hooks; (iv) searching for objects with specific properties or relations to other objects; (v) verification of object's identity and context of it's creation. The presented archivist package extends, in a combination with packages such as knitr and Sweave, the reproducible research paradigm by creating new ways to retrieve and validate previously calculated objects. These new features give a variety of opportunities such as: sharing R objects within reports or articles; adding hooks to R objects in table or figure captions; interactive exploration of object repositories; caching function calls with their results; retrieving object's pedigree (information about how the object was created); automated tracking of the performance of considered models, restoring R libraries to the state in which object was archived.

Tables

Table 1. Table 1: The list of functions available in the \pkg archivist package version 2.1.

	Local	Remote
Repository managment	createLocalRepo
	setLocalRepo	setRemoteRepo
	deleteLocalRepo
	showLocalRepo	showRemoteRepo
	summaryLocalRepo	summaryRemoteRepo
	zipLocalRepo	zipRemoteRepo
	copyLocalRepo	copyRemoteRepo
Artifacts management	saveToLocalRepo
	rmFromLocalRepo
	loadFromLocalRepo	loadFromRemoteRepo
	aread	aread
	asession	asession
	aformat	aformat
	ahistory
	\code%a%
Artifacts’ exploration	searchInLocalRepo	searchInRemoteRepo
	asearch	asearch
	shinySearchInLocalRepo
Extensions	restoreLibs
	atrace
	addHooksToPrint
	createMDGallery

Table 2. Table 2: Tags that are automatically extracted from objects depending on the object’s class. See \code ?Tags for more details.

Artifact’s class	Tags
lm	date, name, class, coefname, rank, df.residual
survfit	date, name, class, strata, type, n, conf.type, conf.int
ggplot	date, name, class, labelx, labely
twins	date, name, class, ac
partition	date, name, class, objective, memb.exp, coeff, k.crisp, conv, clus.avg.widths, avg.width
qda	date, name, class, terms, N, lev, counts, prior, ldet
lda	date, name, class, N, lev, counts, prior, svd
htest	date, name, class, alternative, method, data.name, null.value, statistic, parameter, p.value, intervals, estimate
data.frame	date, name, class, varname
summary.lm	date, name, class, sigma, df, r.squared, adj.r.squared, fstatistic, fstatistic.df
glmnet	date, name, class, dim, nulldev, npasses, offset, nobs
default	date, name, class

Full text

\pkgarchivist: An \proglangR Package for Managing, Recording and Restoring Data Analysis Results

Przemysław Biecek

University of Warsaw \AndMarcin Kosiński

Warsaw University of Technology

Przemyslaw.Biecek@gmail.com

M.P.Kosinski@gmail.com

\Plainauthor

Przemyslaw Biecek, Marcin Kosinski \Plaintitlearchivist: An R Package for Managing, Recording and Restoring Data Analysis Results \ShorttitleR Package \pkgarchivist \Abstract Everything that exists in \proglangR is an object Chambers (2016). This article examines what would be possible if we kept copies of all \proglangR objects that have ever been created. Not only objects but also their properties, meta-data, relations with other objects and information about context in which they were created. We introduce \pkgarchivist, an \proglangR package designed to improve the management of results of data analysis. Key functionalities of this package include: (i) management of local and remote repositories which contain \proglangR objects and their meta-data (objects’ properties and relations between them); (ii) archiving \proglangR objects to repositories; (iii) sharing and retrieving objects (and it’s pedigree) by their unique hooks; (iv) searching for objects with specific properties or relations to other objects; (v) verification of object’s identity and context of it’s creation. The presented \pkgarchivist package extends, in a combination with packages such as \pkgknitr and \pkgSweave, the reproducible research paradigm by creating new ways to retrieve and validate previously calculated objects. These new features give a variety of opportunities such as: sharing \proglangR objects within reports or articles; adding hooks to \proglangR objects in table or figure captions; interactive exploration of object repositories; caching function calls with their results; retrieving object’s pedigree (information about how the object was created); automated tracking of the performance of considered models, restoring \proglangR libraries to the state in which object was archived.

\Keywordsrecordable research, reproducible research, data analysis management, data governance, meta-data, results management \Address Przemysław Biecek

Faculty of Mathematics, Informatics, and Mechanics

University of Warsaw

Banacha 2, 02-097 Warsaw, Poland

E-mail:

 

Marcin Kosiński

Faculty of Mathematics and Information Science

Warsaw University of Technology

Koszykowa 75, 00-662 Warsaw, Poland

E-mail:

1 Introduction

In most of the cases the outcome of the process of data analysis is a set of objects in the form of statistical models, charts or tables. Three requirements are often superimposed to ensure sufficient quality of such results: they should be reproducible, verifiable and accessible. Reproducibility means that there is a process that reproduces results. Verifiability means that it is possible to check whether the newly generated results are identical to previously obtained results, and it is possible to check the context of object’s creation. Accessibility means that results can be easily accessed for future computer based processing. Reproducibility gets increasing attention in the academic literature across various disciplines, see for example Peng (2009) for bioinformatics or Koenker and Zeileis (2009) for the econometric research or Drummond (2009) for more general discussion about differences between replicability and reproducibility.

The \proglangR ecosystem of packages is equipped with wonderful tools such as \pkgknitr (see Xie, 2013, 2015) or \pkgSweave (see Leisch, 2002; Rossini and Leisch, 2003) which allow to create reproducible reports or articles. They follow the literate programming principle, and the R code, its results and its explanations appear together in a single document. It is assumed that the same input and identical instructions executed on the same operating system with the same local settings and with identical versions of installed libraries will result in the same output. Under these assumptions \pkgknitr or \pkgSweave reports are sufficient to recreate the previously obtained results.

But there are cases in which it is not convenient to recreate results from scratch, from raw input. Consider the following situations:

• •

the input data is large or with limited/restricted access (e.g., for genomic data the raw input may easily hit few TB);

• •

computations take a lot of time or require specialized hardware (e.g., calculations tuned for Graphics Processing Unit cards);

• •

calculations are based on a very specific version of software or require commercial versions of software or some functions are deprecated or removed over time. It can be an issue even for open software, e.g., due to rapid development of \proglangR, even widely used packages experience significant changes, like \pkgggplot2 or \pkglme4 in the year 2015;

• •

results are generated and processed periodically and you wish to restore and compare models across all reports.

In such situations it is desirable to retrieve the results that were calculated in the past rather than reproducing them from scratch. Objects that are backed up can be reused even if they cannot be reproduced or the reproducibility will be too complex or time consuming. Alternatively, it may be desired to check whether the reproduced results are the same as those obtained previously.

An interesting example of such a feature are StatLinks (see OECD, 2015) commonly used in reports prepared by OECD (Organization for Economic Co-operation and Development). In addition to scripts that generate results, most tables and plots that are presented in the reports are equipped with their own DOIs (digital object identifier) and web hooks. Through these links readers may download selected tables and plots, in the Excel format. The \codexls and \codexlsx formats are not ideal as they are proprietary and difficult to read in an automated way. But for extensive studies it is convenient and faster to access final results in such formats instead of having scripts that reproduce them.

If the only result from the data analysis is a single plot, a model or a table, it is easy to save it in the \coderda format and make it accessible for the others. But increasing amounts of heterogeneous data results in growing complexity of the process of data analysis. The complexity comes either from data volume, data heterogeneity, numerous steps required for data preparation, results validation etc. Moreover, working with data is often a highly iterative process that generates large amount of partial or final results. For all the above reasons the management of versions of results becomes a task in itself. Neglecting this process results in Reproducibility Debt and may consequently lead to huge additional workload when it comes to recreation of results. The Reproducibility Debt is a part of wider category called Technical Debt (see Sculley et al., 2014).

It should be noted that the concept of recording and exploring relations between objects is not new. Potential applications in auditable data analyses were discussed almost 30 years ago (see Becker and Chambers, 1988). What we present in this article may be perceived as implementation of some of these concepts. It is now easier due to lower costs of data storage.

The \pkgarchivist package helps in managing, sharing, storing, linking and searching for \proglangR objects in a platform agnostic way. Its core functionalities allow for many interesting applications - some of them are presented in the Section 2. The \pkgarchivist package automatically retrieves the object’s meta-data and creates a rich structure that allows for easy management of stored \proglangR objects. The meta-data covers object’s properties such as: name, creation date, class, versions of attached packages, structure and relations between \proglangR objects (as for example, that an object A was used for creation of an object B). All examples presented here are related to \proglangR objects. In the Section 3.4 we discuss how this approach can be extended to other languages.

The rest of the article has the following structure. In the Section 2 (Motivation) we introduce key motivations and use-cases behind \pkgarchivist. In the Section 3 (Functionality) we present all functions available in the package and point out some further directions how this functionality can be integrated with GitHub, \pkgknitr, or be extended on other languages / formats. In the Section 4 (Conclusions) we gather some final thoughts related to recordable and restorable research.

2 Motivation

In this section we present key concepts and some use-cases behind the \pkgarchivist package. In the Section 3 we present all functions available in the \pkgarchivist in a more formal way. First let us introduce some terminology.

• •

Artifact - an \proglangR object that is saved to the repository. Artifacts are identified by their MD5 hashes.

• •

Repository - a collection of artifacts stored as binary files outside of the \proglangR session. Repositories are either local (with a read-write access) or remote (with a read access only). The API for repositories allow for following actions: add, delete, read or search for an artifact with selected Tags. In the current version of the \pkgarchivist local repositories are folders in the file system while remote repositories are Git or Mercurial based repositories. The same mechanism can be used to access repositories pointed as URL addresses or folders attached to \proglangR packages.

• •

MD5 hash - a unique identifier of an artifact. It’s a 32-character-long string, result of cryptographical hash function MD5 (Message Digest algorithm 5). Here, we are using implementation of hash function available in the \pkgdigest package (see Dirk Eddelbuettel et al., 2014). In the \pkgarchivist package MD5 hashes are used as object’s hooks.

• •

Tag - an attribute of an artifact. Tags are represented as character strings; they usually have the following structure: \codekey:value. An artifact may have many tags, even with the same key. Some tags are automatically derived from artifacts, others may be added manually. Tags may be referred as meta-data of artifacts as they describe either properties of artifacts (e.g., class, name, date of creation) or relations between artifacts (e.g., being a part of, being a result of).

The \pkgarchivist package manages \proglangR objects outside the \proglangR session. It stores binary copies of \proglangR objects in \coderda files and provides easy access for seeking and restoring these objects based on timestamps, classes or other properties.

But, why anybody would like to store copies of \proglangR objects? Let’s imagine the following use-cases:

• •

A data scientist creates a report or an article and would like to provide an access to results presented in the article. Typically, these results are presented as plots, tables or models. Apart from including these results in the report or article in a human-readable form, it may be beneficial to be able to restore a given result in a machine-readable form for further processing. Having a possibility to retrieve an \proglangR plot or table, one can perform some further transformation of it. The opportunity to retrieve a regression model enables additional residuals’ validation or applying model to the new data. The \pkgarchivist creates a hook to a copy of \proglangR object which restores the object in a remote \proglangR session. Such hooks are short one-line instructions and can be embedded in figures’ or tables’ captions.

An example report that illustrates this use-case is available at http://bit.ly/1nW9Cvz. A part of it is presented in Figure 1. The report is created with the use of \pkgknitr package. It contains both \proglangR code and it’s results in the form of tables and plots created with \pkgggplot2 package (see Wickham, 2009). In addition, there are also hooks to selected results. These hooks allow to restore a given plot or table directly in the local R session. Hooks of such a form restore a \codegg object in an R session.

\code

archivist::aread("pbiecek/Eseje/arepo/ba7f58fafe7373420e3ddce039558140")

• •

A team of data scientists is working for some time on a forecasting model. During a certain period of time a large set of competing models is created. The team needs a tool that stores all models with additional metadata, such as model performance, information which data was used for model training and testing. The \pkgarchivist creates a shared repository which can be used for storing models along with their meta-data and provides API for searching objects with specific meta-data. The example below reads all objects of the class \codelm, calculates a BIC score for them and sorts objects with respect to these scores.

{Sinput}

R> library("archivist") R> models <- asearch("pbiecek/graphGallery", patterns = "class:lm") R> modelsBIC <- sapply(models, BIC) R> sort(modelsBIC)

{Soutput}

990861c7c27812ee959f10e5f76fe2c3 2a6e492cb6982f230e48cf46023e2e4f 39.05577 67.52735 0a82efeb8250a47718cea9d7f64e5ae7 378237103bb60c58600fe69bed6c7f11 189.73593 189.73593 7f11e03539d48d35f7e7fe7780527ba7 c1b1ef7bcddefb181f79176015bc3931 189.73593 189.73593 0e213ac68a45b6cd454d06b91f991bc7 e58d2f9d50b67ce4d397bf015ec1259c 243.49450 243.49450 18a98048f0584469483afb65294ce3ed 396.16690

• •

Results are generated in a remote \proglangR process, like for example with a Shiny application. The \pkgarchivist saves created \proglangR artifacts in an URL repository.

See for example Figure 2 that presents a screenshot from the Shiny application https://cogito.shinyapps.io/archivistShiny. All plots generated by this application are stored in an \pkgarchivist repository and may be accessed with hooks presented below plots. Following line downloads a single plot directly to the local \proglangR session.

\code

archivist::aread("https://cogito.shinyapps.io/archivistShiny/arepo/

\code

ca680b829abd8f0a4bd2347dcf9fe534").

3 Functionality

The key functionality of the \pkgarchivist package is to manage copies of \proglangR objects, called artifacts, stored as binary files. Artifacts are stored in collections called repositories. Properties of artifacts and relations between artifacts are described by their tags.

Typical lifetime of the repository is presented in Figure 3. The local repository is created with the \codecreateLocalRepo function. It can be set as a default repository so that calls of the other \pkgarchivist functions can be simplified. Once the repository is created, new \proglangR objects can be archived with the \codesaveToLocalRepo function or can be removed with the \codermFromLocalRepo function. Artifacts can be restored from the repository with \codeloadFromLocalRepo function. One can also get all objects that match given criteria with the function named \codesearchInLocalRepo. Both functions have wrappers called \codearead and \codeasearch, respectively, with the simplified and shorter interface. To summarise what kind of artifacts are in the repository one can use \codesummaryLocalRepo or \codeshowLocalRepo functions. The repository can be removed with the \codedeleteLocalRepo function.

Table 1 presents all functions available in the \pkgarchivist package. These functions are divided into four core groups:

• •

Functions for repository management. In this group there are functions used to create a new empty repository, to create a repository as a copy of an existing local or GitHub repository, to backup an entire repository into a single zip file, to present summary statistics of objects stored in the repository and to delete existing repository.

• •

Functions for saving artifacts to a repository, loading artifacts from a repository and removing artifacts from a repository. Functions that show relations between artifacts, present artifacts’ history or context in which they were created.

• •

Functions for searching for artifacts within a repository. Artifacts may be accessed through date of creation, a tag or a list of tags.

• •

Other features that do not fit previous categories.

In sections 3.1-3.4 each group of these functions is presented separately.

3.1 Repository management

A repository is a collection of artifacts and their meta-data. In this section you will find a list of functions for repository management (used to create a new empty repository, create a copy, present summary statistics or delete existing repository).

Technically, repository is a directory with the following structure (see Figure 5).

• •

A \codebackpack.db file which contains an SQLite database. The database contains two tables with a structure presented in Figure 5. The table named artifact contains artifacts’ MD5 hashes and basic information about the artifacts. The table called tag contains artifacts’ tags. Since both artifacts and tags may be added into the database an unspecified number of times, each tag and artifact has one or more time points - one for each attempt to artifact’s or tag’s archiving to the repository.

• •

A subdirectory called \codegallery with artifacts’ storage. Artifacts are stored as separate files. Names of files start with MD5 hashes of corresponding artifacts. Extensions correspond to formats in which artifacts are saved. The current implementation for \proglangR stores artifacts in the \coderda format, but it can be easily extended to handle other formats. Additionally, also an artifact’s miniature is saved. For plots the default format for miniatures is raster file with \codepng extension, for other objects it is a text file with \codetxt extension (e.g., for data frames it contains first few rows).

A repository may be accessed in two ways.

• •

Local - in this case repository is identified by its path in the local file system. The repository is in the read-write mode. If the file system is shared (shared file system on HPC cluster, a Dropbox directory, a mounted folder on Network File System, Secure Shell Filesystem, etc.) multiple users may read and write into the repository at the same time.

• •

Remote - Currently \pkgarchivist supports GitHub and Bitbucket repositories, but it can be easily extended to support any git or mercurial repository, see Section 4. Repository is identified by it’s type (github/bitbucket), a username and the repository’s name. The repository is accessible in read-only mode. Multiple users can read from such repository at the same time. In order to write to a remote repository one should either synchronize a local directory with GitHub/Bitbucket account or use a \pkgarchivist.github package, which is \pkgarchivist’s first extension (see Kosinski and Biecek, 2016).

The logic behind this is as follows. Depending on the user’s needs it is possible to create a single repository per project or per group of projects or keep all artifacts ever created in a single repository. Since (i) a local repository is accessible even without an Internet connection, (ii) the access is faster and (iii) there is both read and write access, it is easier to work with local repositories, which are just a directory identified by its path. If the user wants to share a repository with artifacts with a general public then he or she can publish the local repository on GitHub or Bitbucket or make it available as a subdirectory of an \proglangR package.

3.1.1 Creation of a new empty repository

The \codecreateLocalRepo function creates a new local repository. The \coderepoDir argument points to a directory that will be used as a repository root. The directory will be created if it does not exist. The \codedefault=TRUE argument marks the newly created repository as a default one.

The directory may be specified either by global path or local path. The example below will create a repository named \codearepo in the current working directory.

{Sinput}

R> repo <- "arepo" R> createLocalRepo(repoDir = repo, default = TRUE)

3.1.2 Deletion of an existing repository

The \codedeleteLocalRepo function deletes all artifacts, miniatures, the database with meta-data and the directory identified by the \coderepoDir argument.

{Sinput}

R> repo <- "arepo" R> deleteLocalRepo(repoDir = repo)

3.1.3 Copying artifacts from other repositories

Functions \codecopyLocalRepo and \codecopyRemoteRepo copy selected artifacts from either local or remote (GitHub or Bitbucket) repository into a local repository. Artifacts to be copied are identified by their MD5 hashes.

In the example below the artifact identified by hash \code7f3453331910e3f321ef97d87adb5bad is copied along with its meta-data from remote GitHub repository \codepbiecek/graphGallery to the local repository \codearepo.

{Sinput}

R> repo <- "arepo" R> createLocalRepo(repoDir = repo, default = TRUE) R> copyRemoteRepo(repoTo = repo,

• md5hashes = "7f3453331910e3f321ef97d87adb5bad",
• user = "pbiecek", repo = "graphGallery", repoType = "github")

Functions \codezipLocalRepo and \codezipRemoteRepo download all artifacts and create a single \codezip archive.

3.1.4 Showing repository’s statistics

A repository is a collection of artifacts and their meta-data. Functions \codesummaryLocalRepo and \codesummaryRemoteRepo summarize basic statistics about artifacts in the repository. Functions \codeshowLocalRepo and \codeshowRemoteRepo list all MD5 hashes and artifact’s meta-data.

Functions \codeshow*Repo take argument \codemethod which may be either \code"tags" (the result is a data frame with artifact’s tags) or \code"md5hashes" (default, result is a data frame with artifact’s MD5 hashes).

In the previous example we copied a single artifact from GitHub repository to the local one. The artifact is copied with its tags. In the example below we list all the tags within this single-artifact repository.

{Sinput}

R> showLocalRepo(repoDir = repo, method = "tags")

{Soutput} artifact tag createdDate 1 7f3453331910e3f321ef97d87adb5b format:rda 2016-02-09 14:37:06 2 7f3453331910e3f321ef97d87adb5b class:gg 2016-02-09 14:37:06 3 7f3453331910e3f321ef97d87adb5b class:ggplot 2016-02-09 14:37:06 4 7f3453331910e3f321ef97d87adb5b labelx:Sepal.Length 2016-02-09 14:37:06 5 7f3453331910e3f321ef97d87adb5b labely:Petal.Length 2016-02-09 14:37:06 6 7f3453331910e3f321ef97d87adb5b date:2016-02-09 14:37:06 2016-02-09 14:37:06 7 7f3453331910e3f321ef97d87adb5b format:png 2016-02-09 14:37:06 8 ff575c261c949d073b2895b05d1097 relationWith:2166d…… 2015-06-22 17:17:14 8 ff575c261c949d073b2895b05d1097 sessionInfo:3b8c60…… 2015-06-22 17:17:14

In the example below the function \codesummaryLocalRepo is used to list summaries of artifacts in the repository called \codegraphGallery which is attached to the \pkgarchivist package. One can find information about dates on which artifacts were added, classes of artifacts and the total number of artifacts in the repository.

{Sinput}

R> summaryLocalRepo(repoDir =

• system.file("graphGallery", package = "archivist"))

{Soutput} Number of archived artifacts in Repository: 7 Number of archived datasets in Repository: 3 Number of various classes archived in Repository: Number gg 2 ggplot 2 lm 3 data.frame 2 summary.lm 1 Saves per day in Repository: Saves 2016-02-07 6 2016-02-08 15

3.1.5 Setting a default repository

In most of the cases we work with one repository per project. In such cases it is convenient to set a default local or remote repository. It can be done with \codesetLocalRepo or \codesetRemoteRepo functions. Look at the example below.

{Sinput}

R> setRemoteRepo(user = "pbiecek", repo = "graphGallery", repoType = "github") R> setLocalRepo(repoDir =

• system.file("graphGallery", package = "archivist"))

After setting a default repository, one can use the following functions

• •

\code

saveToLocalRepo,

• •

\code

loadFromLocalRepo, \codeloadFromRemoteRepo,

• •

\code

rmFromLocalRepo,

• •

\code

searchInLocalRepo, \codesearchInRemoteRepo,

without specification of \coderepoDir or \codeuser/repo/branch/subdir/repoType arguments.

For example, the instruction below will add \codeiris data frame to the default local repository.

{Sinput}

R> setLocalRepo(repoDir = repo) R> data("iris") R> saveToRepo(iris)

Another option for setting a default value for an argument is the function \codeaoptions(). It sets the default value for any argument that is used by \codearchivist. For example the instruction below sets the default value for \coderepoType to \code"github". {Sinput} R> aoptions("repoType", "github")

3.2 Artifact management

An artifact is an \proglangR object with its meta-data. Artifacts are stored in repositories. Key functions for artifact’s management are functions for saving, loading and removing artifacts from a repository.

3.2.1 Saving an \proglangR object into a repository

The \codesaveToLocalRepo function saves any \proglangR object into the selected repository. It stores in the repository both the object and its tags. Some tags and some meta-data are extracted in an automated way. The \codesaveToLocalRepo function recognizes the class of the artifact and extracts tags typical for that class. It is possible to add support for a new class of objects or change list of tags extracted for selected classes, just extend the generic function \codeextractTags(). Table 2 lists classes that are recognized in the current version of the package and lists tags that are derived automatically from objects of a given class. For other classes the following attributes are extracted: name, creation time and MD5 hash.

The \codesaveToLocalRepo function takes at least two arguments: \codeartifact - an \proglangR object which is about to be saved and \coderepoDir which is a path to the local repository. The process of adding an \proglangR object to the repository triggers a chain of actions listed below. By setting some arguments of \codesaveToLocalRepo to \codeFALSE some of these actions may be skipped.

• •

The name of the object is derived and stored as the object’s tag \codename:xxx. It may be useful when searching for an object. One can search for all objects that had a specific name with \codeasearch(pattern="name:iris").

• •

An MD5 hash is calculated for the object with the use of \pkgdigest package. Then the object is saved as a binary file named \codemd5hash.rda with the use of \codesave function.

• •

If there is any dependent object, it is saved separately to the repository (e.g., for object of class \codegg or \codelm the \codedata slot is extracted from the object and saved separately. Additionally a tag \coderelationWith:xxx is added, where \codexxx is the MD5 hash of the dataset).

• •

The current session info, with the list of versions of attached packages, is saved to the repository. The session info is linked to the artifact. The link is a tag of the form \codesessionInfo:xxx, where \codexxx stands for MD5 hash of the object with session info.

• •

A set of tags is extracted automatically and these tags are saved to the repository. See Table 2 for the list of tags that are automatically derived. Tags extracted for a given class are defined by the generic \codeextractTags function.

• •

Additional tags specified by a user (with the \codeuserTags argument) are saved to the repository as well.

• •

A miniature for the object is created – for plots it is a png file while for data frames or models it is a text description of the object.

The following example creates a plot of the class \codegg and saves the object into the repository. Plots created with the use of \pkgggplot2 package are objects and can be serialized in the same way as any other \proglangR objects (see Wickham, 2009). A hash of the recorded object is returned. In the example below it is \code11127cc6ce69a89d11d0e30865a33c13. By default, the related data object is also saved. In this case the dependent object is a dataset \codeiris which is saved with the hash \codeff575c261c949d073b2895b05d1097c3. {Sinput} R> library("ggplot2") R> repo <- "arepo" R> pl <- qplot(Sepal.Length, Petal.Length, data = iris) R> saveToLocalRepo(pl, repoDir = repo)

{Soutput} [1] "11127cc6ce69a89d11d0e30865a33c13" attr(,"data") [1] "ff575c261c949d073b2895b05d1097c3"

The function \codesaveToLocalRepo extracts additional tags such as the name of the original object (here: \codename:pl), its class (\codeclass:gg), labels on OX and OY axes (\codelabelx:Sepal.Length) and MD5 hash of the data object. These tags are listed if we use \codeshowLocalRepo function on the repository.

{Sinput}

R> showLocalRepo(repoDir = repo, "tags")

{Soutput} artifact tag createdDate 1 11127cc6ce69a89d11d0e30865a33c format:rda 2016-02-09 16:42:59 2 11127cc6ce69a89d11d0e30865a33c name:pl 2016-02-09 16:42:59 3 11127cc6ce69a89d11d0e30865a33c class:gg 2016-02-09 16:42:59 4 11127cc6ce69a89d11d0e30865a33c class:ggplot 2016-02-09 16:42:59 5 11127cc6ce69a89d11d0e30865a33c labelx:Sepal.Length 2016-02-09 16:42:59 6 11127cc6ce69a89d11d0e30865a33c labely:Petal.Length 2016-02-09 16:42:59 7 11127cc6ce69a89d11d0e30865a33c date:20160209 16:42:59 2016-02-09 16:42:59 8 11127cc6ce69a89d11d0e30865a33c session_info:e0373….. 2016-02-09 16:42:59 9 11127cc6ce69a89d11d0e30865a33c format:png 2016-02-09 16:42:59 10 e037375a5f757efcc28561c0a1a2ef format:rda 2016-02-09 16:42:59 11 ff575c261c949d073b2895b05d1097 format:rda 2016-02-09 16:42:59 12 ff575c261c949d073b2895b05d1097 session_info:e0373….. 2016-02-09 16:42:59 13 ff575c261c949d073b2895b05d1097 format:txt 2016-02-09 16:42:59 14 ff575c261c949d073b2895b05d1097 relationWith:11127….. 2016-02-09 16:42:59

By default, for each artifact also it’s context, i.e., session info, is saved. It can be accessed with the function \codeasession(). See the example below. Such additional information may be very useful if we cannot replicate previous results and we are in the need of recovering the exact versions of important packages, which can be done with \coderestoreLibs function.

{Sinput}

R> asession("11127cc6ce69a89d11d0e30865a33c13")

{Soutput} Session info —————————————————— setting value version R version 3.2.2 (2015-08-14) system x86_64, darwin13.4.0 ui RStudio (0.99.441) Packages ———————————————————- package * version date source acepack 1.3-3.3 2013-05-03 CRAN (R 3.1.0) archivist * 1.9.7.3 2016-02-09 CRAN (R 3.2.2) ggplot2 * 2.0.0 2015-12-16 Github (hadley/ggplot2@11679cd) gridExtra * 2.0.0 2015-07-14 CRAN (R 3.2.0) …

3.2.2 Serialization of an object creation event into repository

The \pkgarchivist provides a new operator \code%a% that works as the extended pipe operator \code%>% from the \pkgmagrittr package (see Bache and Wickham, 2014, for more details). In addition, it saves the resulting object to the default \pkgarchivist repository together with the function call and its parameters. The default repository should be set first, see the \codesetLocalRepo function for instructions how to do this. With this functionality it is possible to trace function calls and extract pedigree for some artifacts.

{Sinput}

R> library("archivist") R> createLocalRepo("arepo", default = TRUE) R> library("dplyr") R> iris + dpyr::filter(Sepal.Length < 6) + lm(Petal.Length Species, data=.) + summary() -> tmp

How to recreate an object’s history? The function \codeahistory extracts the chain of calls that leads to the selected object. As an argument one can specify either an object’s value or its MD5 hash. The value of \codeahistory function is a \codedata.frame with two columns – first contains function calls while second contains MD5 hashes of partial results.

In the example above, a chain of three operations converts input \codeiris data into the \codetmp object. The \pkgdplyr package (see Wickham and Francois, 2015) has to be loaded first since the function \codefilter is used in this example. Following lines present the chain of consecutive transformations that are recorded in the repository.

{Sinput}

R> ahistory(tmp) R> ahistory(md5hash = "050e41ec3bc40b3004bc6bdd356acae7")

{Soutput} iris [ff575c261c949d073b2895b05d1097c3] -> filter(Sepal.Length < 6) [d3696e13d15223c7d0bbccb33cc20a11] -> lm(Petal.Length   Species, data = .) [990861c7c27812ee959f10e5f76fe2c3] -> summary() [050e41ec3bc40b3004bc6bdd356acae7]

In order to restore an object’s pedigree all partial results must be saved in a repository. So this option will work only for objects created by a chain of calls that use the \code%a% operator.

3.2.3 Loading an object from a repository

To read an object from repository we may consider the following four scenarios.

• •

We know the object’s MD5 hash and the object is in a local directory.

• •

We know the object’s MD5 hash and the object is in a remote repository, i.e., on GitHub or BitBucket.

• •

We do not know the hash but we know some properties of the object so we need to find it first by its tags. The object is in a local repository.

• •

As above, but the object is in a remote repository.

If we know the MD5 hash of the requested artifact, we can directly load the object from the repository and in this section we are going to show how this can be done. If we do not know the MD5 hash, then we need to use one of \codesearch* functions presented in Section 3.3.

Functions \codeloadFromLocalRepo and \codeloadFromRemoteRepo read artifacts from either local or remote repositories. The local repository is defined by a path to it’s root; remote repository is defined by it’s type (currently \code"github" (default) or \code"bitbucket"), the username, repository’s name and a subdirectory within the repository. In both functions the argument \codevalue specifies whether the function should return the object by value (\codevalue=TRUE) or it should load the object into the namespace with its original name (\codevalue=FALSE).

For the purpose of this example we have created a repository \codegraphGallery, with two objects: a plot and a regression model. The repository is available both on GitHub (see \codehttps://github.com/pbiecek/graphGallery) and within the \pkgarchivist package (see the \codegraphGallery directory). Two archived objects have \code7f3453331910e3f321ef97d87adb5bad and \code2a6e492cb6982f230e48cf46023e2e4f hashes respectively.

The full MD5 hash of an artifact is a 32-characters-long string but it is enough to set only the first few characters. In the example below it is enough to use \code"7f34533" prefix to load an artifact with the \code"7f3453331910e3f321ef97d87adb5bad" hash. There is only one artifact with prefix \code"7f34533" in its MD5 hash. If there is more, all that match the prefix are returned. Note that one should not use this feature unless is sure that new objects with colliding hashes will not be added. For small repositories conflicts are unlikely even for first five characters, but be careful when using this feature.

Both following instructions retrieve an \proglangR object from GitHub, load it into \proglangR session and make it accessible for further processing. In this case it is a \codeggplot2 object so after being loaded the \codeprint function is triggered and a plot is generated (see Figure 6). Note that by default the GitHub is assumed, but this may be changed with the parameter \coderepoType.

{Sinput}

R> loadFromRemoteRepo("7f3453331910e3f321ef97d87adb5bad",

• repo = "graphGallery", user = "pbiecek", value = TRUE) R> loadFromLocalRepo("7f34533",
• system.file("graphGallery", package = "archivist"), value = TRUE)

The \codearead function is a wrapper over \codeloadFromRemoteRepo with more compact form. Shorter instructions and shorter code snippets might be placed in a figure or table caption. The single line below reads an object with the \code7f34533… hash from \codegraphGallery GitHub repository that is owned by the \codepbiecek user.

{Sinput}

R> archivist::aread("pbiecek/graphGallery/7f3453331910e3f321ef97d87adb5bad")

The following instructions retrieve the same \proglangR object but this time from the \codegraphGallery repository attached to the \pkgarchivist package. Note that the default repository is set first with the \codesetLocalRepo function.

{Sinput}

R> library("archivist") R> setLocalRepo(system.file("graphGallery", package = "archivist")) R> aread("7f3453331910e3f321ef97d87adb5bad")

The use of MD5 hashes as objects identifiers has some advantages. In some use cases we may be restricted to use only models approved by some authority. For example, due to some hypothetical regulatory issues in production it might be advisable to use only a specific version of a model (such as credit scoring model or some forecasting model).

In the \pkgarchivist package all objects have their cryptographical hash calculated with the MD5 algorithm. One can use the \codedigest function to validate the object’s MD5 hash at any moment. One can also call an object the from repository by its MD5 hash. Having a list of MD5 hashes of allowed objects one can validate their identity.

In the example below the downloaded regression model is digested to confirm its identity.

{Sinput}

R> setLocalRepo(system.file("graphGallery", package = "archivist")) R> model <- aread("2a6e492cb6982f230e48cf46023e2e4f") R> digest::digest(model)

{Soutput} "2a6e492cb6982f230e48cf46023e2e4f"

3.2.4 Removal of an object from a repository

To remove an artifact from a repository one can use the \codermFromLocalRepo function.

In the example below the artifact \code92ada1e052d4d963e5787bfc9c4b506c and all its tags are removed from the repository called \coderepo.

{Sinput}

R> rmFromLocalRepo("7f3453331910e3f321ef97d87adb5bad", repoDir = repo)

A list of artifact’s hashes that should be removed may be obtained with the \codesearch* function. The example below searches for all artifacts older than 30 days and removes them from the \coderepo repository.

{Sinput}

R> obj2rm <- searchInLocalRepo(list(dateFrom = "2010-01-01",

• dateTo = Sys.Date()-30), repoDir = repo) R> rmFromLocalRepo(obj2rm, repoDir = repo, many = TRUE)

It is also possible to remove many artifacts with one call. Broader examples of this function are explained in the package manual page accessed from \proglangR with \code?rmFromLocalRepo.

3.3 Search for an artifact and explore the repository

One of the advantages of the \pkgarchivist package is the automated derivation of artifact’s tags and meta-data. It is useful when one wants to find previously calculated results in a large collection of \proglangR objects. Relations between artifacts are useful when we want to process the structure dependencies between artifacts. Below we present a list of functions for searching for artifacts on the basis of their properties.

3.3.1 Search in a local or remote repository

If we do not know the MD5 hashes of artifacts that are of our interest, we can find them with the use of \codesearch* functions.

Searching within a local repository and a remote repository is very similar. Functions \codesearchInLocalRepo or \codesearchInRemoteRepo differ only in the way in which the repository is specified.

In both functions the \codepattern argument may be either a tag (name, class, varname or other) or a date period in which given artifact was created. Hashes of all artifacts that meet all criteria (i.e., were created within a given time interval or have a given tag attached) are returned.

For example, the following command retrieves MD5 hashes of all objects of the class \codegg from the \codepbiecek/graphGallery repository.

{Sinput}

R> searchInLocalRepo(pattern = "class:gg",

• repoDir = system.file("graphGallery", package = "archivist"))

{Soutput} [1] "7f3453331910e3f321ef97d87adb5bad" "369227e67f9164dcbe934dadf2b53cc2"

To get a list of artifacts created within a given date range one can use following instruction.

{Sinput}

R> searchInLocalRepo(pattern = list(dateFrom = "2016-01-01",

• dateTo = "2016-02-07"),
• repoDir = system.file("graphGallery", package = "archivist"))

{Soutput} [1] "d9313a0de3e2980201a8971e3384ff26" "ff575c261c949d073b2895b05d1097c3" [3] "2a6e492cb6982f230e48cf46023e2e4f" "93ecfdf1436932e2860c6dbdf2abc2ad" [5] "afb2550d0f886f0cf3b050f04c5cd4f8"

The \codesearchInLocalRepo and \codesearchInRemoteRepo functions allow to use more than one searching criteria. Additional argument \codeintersect specifies if the resulting objects have to met all or any of the search criteria.

{Sinput}

R> searchInLocalRepo(pattern=c("class:gg", "labelx:Sepal.Length"),

• repoDir = system.file("graphGallery", package = "archivist"))

{Soutput} [1] "369227e67f9164dcbe934dadf2b53cc2" "7f3453331910e3f321ef97d87adb5bad"

These two functions return MD5 hashes of artifacts. In order to load these artifacts from repository one needs to use either \codeloadFrom*Repo or \codearead functions. Since both operations are usually performed together (search for MD5 hashes of artifacts by their tag / load artifacts with given MD5 hashes), one can use the \codeasearch function which retrieves MD5 hashes and returns a list with values of artifacts that meet all selected criteria.

3.3.2 Retrieval of a list of \proglangR objects with given tags

When working in a team or for a longer period of time, one produces a lot of partial results and it becomes harder and harder to trace what kind of analyses were conducted in the past and where are the results.

The \pkgarchivist extracts meta-data from \proglangR objects in the very same moment they are archived in a repository. For many researchers objects are so valuable, due to their pedigree and meta-data, that they can be regarded as artifacts. Having such additional meta-data it is easier to search for previously generated partial results, e.g., by specifying what kind of model with which variables we are looking for.

For example, the code below retrieves all objects of the \codelm class with the \codeSepal.Length variable from within a list of dependent variables. In this repository only two artifacts (here \codelm models) match both conditions.

The following instruction searches within the default local repository.

{Sinput}

R> setLocalRepo(system.file("graphGallery", package = "archivist")) R> models <- asearch(patterns = c("class:lm", "coefname:Sepal.Length"))

Below is the code that searches within the GitHub repository.

{Sinput}

R> models <- asearch("pbiecek/graphGallery",

• patterns = c("class:lm", "coefname:Sepal.Length")) R> lapply(models, coef)

{Soutput} $`18a98048f0584469483afb65294ce3ed`(Intercept)Sepal.Length-7.1014431.858433\par$‘2a6e492cb6982f230e48cf46023e2e4f‘ (Intercept) Sepal.Length Speciesversicolor Speciesvirginica -1.7023422 0.6321099 2.2101378 3.0900021

The following instruction retrieves all artifacts of the \codegg class (created with the package \pkgggplot2) with label \codeSepal.Length on the X axis. Two objects are returned as a result. They are plotted together by the \codegrid.arrange function from \pkggridExtra package (see Auguie, 2015).

{Sinput}

R> plots <- asearch(patterns = c("class:gg", "labelx:Sepal.Length")) R> length(plots)

{Soutput} [1] 2

{Sinput} R> library("gridExtra") R> do.call(grid.arrange, plots)

Result of these instructions is presented in Figure 8.

3.3.3 Interactive search in a local repository

For local repositories, it is also possible to explore the repository interactively with the \codeshinySearchInLocalRepo function. This function launches a Shiny application (see Chang et al., 2015) which is dynamically created and which allows for interactive specification of tags and sorting criteria. See Figure 8 with an example screenshot of this application.

In the text box area one can specify tags that filter out objects presented on the right panel. Only miniatures of objects that meet all these criteria are presented. Additionally, the instruction \codesort:key sorts the artifacts along the key. For example, use \code"sort:createdDate" to sort miniatures along the date of creation of the object.

{Sinput}

R> arepo <- system.file("graphGallery", package = "archivist") R> shinySearchInLocalRepo(arepo)

3.4 Extensions

The \pkgarchivist package is designed as a multi-purpose manager of objects. In this section we present some specific extensions.

3.4.1 Archiving all results of a specific function

The \codetrace() function from the \pkgbase package allows to insert a specific instruction to the body of a selected function. It can be used for example to call \codesaveToLocalRepo() function at the end of a selected function.

In the example below we modify the \codelm() function so that after it’s each execution the created \codelm model is automatically added to the default local repository \codeallModels.

{Sinput}

R> library("archivist") R> createLocalRepo("allModels", default = TRUE) R> atrace("lm", "z") Tracing function "lm" in package "stats" R> lm(Sepal.Length   Sepal.Width, data=iris) Tracing lm(Sepal.Length   Sepal.Width, data = iris) on exit

Call: lm(formula = Sepal.Length   Sepal.Width, data = iris)

Coefficients: (Intercept) Sepal.Width 6.5262 -0.2234

R> sapply(asearch("class:lm"), BIC) 42fcf77af2c40f70c445cbba513aeabd 381.0236

3.4.2 Integration with the \pkgknitr package

The \pkgknitr package is a tool that transforms a mixture of \proglangR code and descriptions in natural language into a md, html or pdf report. Moreover the produced report contains results generated by the included \proglangR code. On one hand reader knows that presented results are generated by presented code. On the second hand the author does not waste time on coping the results, since they are automatically included in the output. Results included in a report are usually plots or tables. In such form they cannot be loaded from the pdf/html file directly to \proglangR. The \pkgarchivist package records objects and makes them easier to access through local, GitHub or BitBucket repositories.

The function \codeaddHooksToPrint combines these two tools. A call to this function should be included on the beginning of a \pkgknitr report. It creates a new generic \codeprint functions for classes specified by the \codeclass argument. These functions save objects to the repository and add corresponding hooks to the report after every attempt to \codeprint the object. Hooks are short instructions on how the recorded objects can be accessed.

An example is presented in the report http://bit.ly/1nW9Cvz. Part of this report is presented in Figure 1. On the beginning there is a snippet presented below. It automatically adds hooks to the html report for all objects of classes \codeggplot or \codedata.frame.

{Sinput}

R> addHooksToPrint(class=c("ggplot", "data.frame"),

• repoDir = "arepo",
• repo = "Eseje", user = "pbiecek", subdir = "arepo")

As a result, just before each plot, there are automatically created hooks to corresponding objects e.g., \codearchivist::aread("pbiecek/Eseje/arepo/24ea7c04b861083d4bf56eee1c5a17b7"). These hooks serve also as links to the corresponding \proglangR objects.

The biggest advantage of this integration is that a single call to \codeaddHooksToPrint is needed to enrich the \pkgknitr report in \pkgarchivist hooks for all interesting objects.

3.4.3 Gallery of artifacts in the repository

Information about artifacts is stored in an SQLite database in the \codebackpack.db file. The \codecreateMDGallery function creates a single markdown file with gallery of all artifacts in the repository.

Such gallery, if saved as file named \codereadme.md, will automatically list all artifacts with miniatures and tags in the GitHub web portal user interface. See an example gallery at http://bit.ly/1Q62Tpz. This gallery was created with the following instruction. A part of the result is presented in Figure 9.

{Sinput}

R> createMDGallery("arepo/readme.md",

• repo = "Eseje", user = "pbiecek", subdir = "arepo",
• addMiniature = TRUE, addTags = TRUE)

3.4.4 Support for other repositories, other languages and other formats

The current implementation of \pkgarchivist supports local, GitHub and BitBucket repositories. The package is implemented in \proglangR and saves artifacts in the \coderda format.

In order to support other repositories one can extend the function \codegetRemoteHook. It is used internally by other \pkgarchivist functions to generate URL addresses to files in remote repositories. In order to support other repositories it’s enough to extend this function.

All metadata related to artifacts is sorted in an SQLite database in \codebackpack.db file. This database can be accessed from other languages. Objects are stored as files and can be added in different formats. Each artifact has an additional tag \codeformat:xxx that specifies in which format the artifact is saved, one artifact can be saved in more than one format. Currently artifacts are stored as \coderda files. In order to save objects in other formats, like \codejson or \codecsv, it is enough to extend the \codesaveToLocalRepo function. In order to load objects from other formats it is enough to overload \codeloadFromLocalRepo and \codeloadFromRemoteRepo functions.

3.4.5 Restoring older versions of packages

In some cases, in order to use an artifact it is not enough to restore it. A good example of this problem are objects of the \codegg class created with \pkgggplot2 package. The structure of \codegg objects is different in package \pkgggplot2 in the version 1.0, different in the version 2.0 and different in the version 2.1. It means that even if we have restored an object that was created with package in version 2.0 we will not be able to use the \codeplot function for this object if one uses \pkgggplot2 package in the version 2.1 nor 1.0.

To use the object we need to downgrade \pkgggplot2 package to the version 2.0. This is possible with the \coderestoreLibs function. For a given hash of an artifact the \coderestoreLibs function restores it’s \codesession_info and reinstalls required packages with versions attached during the artifact’s archiving. Packages can be reinstalled in the new directory, not to affect the default \proglangR libraries.

For example, the \code600bda83cb840947976bd1ce3a11879d object was created with \pkgggplot2 version 2.0. The \codeasession() function checks versions of packages that were then attached.

{Sinput}

R> asession("pbiecek/graphGallery/arepo/600bda83cb840947976bd1ce3a11879d")

{Soutput} … Formula 1.2-1 2015-04-07 CRAN (R 3.1.3) ggplot2 2.0.0 2015-12-16 Github (hadley/ggplot2@11679cd) gridExtra * 2.0.0 2015-07-14 CRAN (R 3.2.0) …

Here the \pkgggplot2 was in the version 2.0 and was installed from GitHub. The \coderestoreLibs() function reinstalls all libraries from proper repositories (here GitHub) to proper versions (here commit \code11679cd).

{Sinput}

R> restoreLibs("pbiecek/graphGallery/arepo/600bda83cb840947976bd1ce3a11879d")

After that one can load and plot the \codeggplot object since the structure of \codegg object is compatible with installed libraries.

{Sinput}

R> aread("pbiecek/graphGallery/arepo/600bda83cb840947976bd1ce3a11879d")

4 Conclusions

The goal of a data analysis is not only to answer a research question based on data but also to collect findings that support that answer. These findings usually take the form of a table, plot or regression/classification model and are usually presented in articles or reports. Such objects are mostly well presented graphically, but they are hard to recreate back in a computer.

In this paper we have presented the \proglangR package called \pkgarchivist, which implements the logic of recordable research. The \pkgarchivist stores \proglangR objects in repositories. The data scientist may share obtained results with other users, create hooks to models and then embed these hooks in articles, reports or web applications. One may also search within a repository and look for artifacts with given properties or relations with other artifacts. One may also validate the object’s identity or derive its pedigree.

Repositories may be shared among team members or between different computers or systems. Statistical models or plots may be stored in a single repository which simplifies the object management.

In this article we have also presented some use-cases for the \pkgarchivist package, such as: hooks for \proglangR objects that can be embedded in reports or articles, interactive searching within repository or retrieving object’s pedigree.

5 Acknowledgments

Thanks go to Ross Ihaka, Łukasz Bartnik, Cezary Chudzian and two anonymous reviewers for valuable discussions and comments on the idea of recordable research and early versions of this paper. We would like to thank Witold Chodor for his great contributions to the development of this package. The package \pkgarchivist was initiated as an open project in the company iQor Polska sp. z o.o..