A Workflow Manager for Complex NLP and Content Curation Pipelines

TL;DR

This paper introduces a workflow manager designed to facilitate the creation and customization of complex NLP pipelines, emphasizing interoperability, scalability, and efficiency for real-world industrial applications.

Contribution

It presents a novel workflow management system with a custom language and architecture tailored for flexible, scalable NLP processing in industry settings.

Findings

System implementation based on real-world use cases

Supports diverse NLP tasks and hardware configurations

Enhances interoperability and resource management

Abstract

We present a workflow manager for the flexible creation and customisation of NLP processing pipelines. The workflow manager addresses challenges in interoperability across various different NLP tasks and hardware-based resource usage. Based on the four key principles of generality, flexibility, scalability and efficiency, we present the first version of the workflow manager by providing details on its custom definition language, explaining the communication components and the general system architecture and setup. We currently implement the system, which is grounded and motivated by real-world industry use cases in several innovation and transfer projects.