Corpus Conversion Service: A machine learning platform to ingest documents at scale [Poster abstract]

TL;DR

This paper introduces a scalable machine learning platform for ingesting and converting large volumes of scientific and technical documents into structured data, overcoming challenges posed by diverse formats and complex layouts.

Contribution

It presents a novel platform that enables training custom ML models for document conversion at scale, achieving high precision and recall.

Findings

Precision/recall over 97% for document conversion

Scalable microservices architecture demonstrated

Effective training of custom models on document collections

Abstract

Over the past few decades, the amount of scientific articles and technical literature has increased exponentially in size. Consequently, there is a great need for systems that can ingest these documents at scale and make their content discoverable. Unfortunately, both the format of these documents (e.g. the PDF format or bitmap images) as well as the presentation of the data (e.g. complex tables) make the extraction of qualitative and quantitive data extremely challenging. We present a platform to ingest documents at scale which is powered by Machine Learning techniques and allows the user to train custom models on document collections. We show precision/recall results greater than 97% with regard to conversion to structured formats, as well as scaling evidence for each of the microservices constituting the platform.