Structured references from PDF articles: assessing the tools for bibliographic reference extraction and parsing

TL;DR

This paper evaluates seven tools for extracting and parsing bibliographic references from PDF articles, comparing their performance across diverse subject areas to identify the most effective solutions.

Contribution

It provides a comprehensive comparison and assessment of existing reference extraction tools from PDFs, highlighting their strengths and weaknesses.

Findings

Anystyle achieved the highest overall performance.

Cermine was the second-best tool overall.

Different tools excelled in specific subject areas.

Abstract

Many solutions have been provided to extract bibliographic references from PDF papers. Machine learning, rule-based and regular expressions approaches were among the most used methods adopted in tools for addressing this task. This work aims to identify and evaluate all and only the tools which, given a full-text paper in PDF format, can recognise, extract and parse bibliographic references. We identified seven tools: Anystyle, Cermine, ExCite, Grobid, Pdfssa4met, Scholarcy and Science Parse. We compared and evaluated them against a corpus of 56 PDF articles published in 27 subject areas. Indeed, Anystyle obtained the best overall score, followed by Cermine. However, in some subject areas, other tools had better results for specific tasks.