Collage: Decomposable Rapid Prototyping for Information Extraction on Scientific PDFs

TL;DR

Collage is a versatile tool that enables rapid prototyping, visualization, and evaluation of various information extraction models on scientific PDFs, facilitating comparison, debugging, and understanding of NLP systems in scientific domains.

Contribution

The paper introduces Collage, a flexible platform for testing and analyzing multiple information extraction models on scientific PDFs, with features for visualization and debugging.

Findings

Supports HuggingFace token classifiers and LLMs out of the box

Provides detailed views of intermediate processing states

Facilitates comparison and debugging of models on scientific PDFs

Abstract

Recent years in NLP have seen the continued development of domain-specific information extraction tools for scientific documents, alongside the release of increasingly multimodal pretrained transformer models. While the opportunity for scientists outside of NLP to evaluate and apply such systems to their own domains has never been clearer, these models are difficult to compare: they accept different input formats, are often black-box and give little insight into processing failures, and rarely handle PDF documents, the most common format of scientific publication. In this work, we present Collage, a tool designed for rapid prototyping, visualization, and evaluation of different information extraction models on scientific PDFs. Collage allows the use and evaluation of any HuggingFace token classifier, several LLMs, and multiple other task-specific models out of the box, and provides extensible software interfaces to accelerate experimentation with new models. Further, we enable both developers and users of NLP-based tools to inspect, debug, and better understand modeling pipelines by providing granular views of intermediate states of processing. We demonstrate our system in the context of information extraction to assist with literature review in materials science.