Toward Reliable Ad-hoc Scientific Information Extraction: A Case Study on Two Materials Datasets

TL;DR

This paper investigates GPT-4's capability to perform ad-hoc, schema-based information extraction from scientific literature in materials science, analyzing its accuracy and limitations through expert error analysis.

Contribution

It introduces a case study evaluating GPT-4's effectiveness in replicating existing datasets and provides insights for improving automated scientific information extraction.

Findings

GPT-4 can partially replicate datasets with basic prompting

Manual error analysis reveals specific extraction challenges

Research directions proposed for enhancing extraction fidelity

Abstract

We explore the ability of GPT-4 to perform ad-hoc schema based information extraction from scientific literature. We assess specifically whether it can, with a basic prompting approach, replicate two existing material science datasets, given the manuscripts from which they were originally manually extracted. We employ materials scientists to perform a detailed manual error analysis to assess where the model struggles to faithfully extract the desired information, and draw on their insights to suggest research directions to address this broadly important task.