A Data-Centric Framework for Composable NLP Workflows

TL;DR

This paper introduces an open-source, modular framework that enables the rapid development of complex, composable NLP workflows with a unified data representation and extensive interoperability across components.

Contribution

It presents a unified, extensible framework with a common data format and a large set of processors, facilitating flexible and efficient construction of NLP pipelines.

Findings

Supports heterogeneous NLP components with a unified data format

Enables easy assembly and extension of NLP workflows

Provides visualization and annotation tools for practical applications

Abstract

Empirical natural language processing (NLP) systems in application domains (e.g., healthcare, finance, education) involve interoperation among multiple components, ranging from data ingestion, human annotation, to text retrieval, analysis, generation, and visualization. We establish a unified open-source framework to support fast development of such sophisticated NLP workflows in a composable manner. The framework introduces a uniform data representation to encode heterogeneous results by a wide range of NLP tasks. It offers a large repository of processors for NLP tasks, visualization, and annotation, which can be easily assembled with full interoperability under the unified representation. The highly extensible framework allows plugging in custom processors from external off-the-shelf NLP and deep learning libraries. The whole framework is delivered through two modularized yet integratable open-source projects, namely Forte (for workflow infrastructure and NLP function processors) and Stave (for user interaction, visualization, and annotation).