# Koji: Automating pipelines with mixed-semantics data sources

## TL;DR

Koji introduces a unified, result-oriented semantic framework for defining reproducible data processing workflows that seamlessly integrate files and services, automating control-plane concerns and enabling simple, declarative pipeline specifications.

## Contribution

It presents a novel semantic approach for workflows manipulating mixed-semantics data sources, with a declarative configuration mechanism for reproducible data pipelines.

## Key findings

- Unified treatment of files and services simplifies integration.
- Automates failure recovery and computation reuse.
- Supports declarative, reproducible pipeline definitions.

## Abstract

We propose a new result-oriented semantic for defining data processing workflows that manipulate data in different semantic forms (files or services) in a unified manner. This approach enables users to define workflows for a vast variety of reproducible data-processing tasks in a simple declarative manner which focuses on application-level results, while automating all control-plane considerations (like failure recovery without loss of progress and computation reuse) behind the scenes. The uniform treatment of files and services as data enables easy integration with existing data sources (e.g. data acquisition APIs) and sinks of data (e.g. database services). Whereas the focus on containers as transformations enables reuse of existing data-processing systems. We describe a declarative configuration mechanism, which can be viewed as an intermediate representation (IR) of reproducible data processing pipelines in the same spirit as, for instance, TensorFlow\cite{tensorflow} and ONNX\cite{onnx} utilize IRs for defining tensor-processing pipelines.