Mathematical Derivation Graphs: A Relation Extraction Task in STEM Manuscripts

TL;DR

This paper introduces the Mathematical Derivation Graphs Dataset to analyze mathematical relationships in STEM texts, evaluating NLP and ML models' ability to extract derivation dependencies with moderate success.

Contribution

It presents the first dataset and evaluation framework for extracting derivation relationships between equations in STEM articles, expanding NLP applications to mathematical content.

Findings

Best LLMs achieve 45-52% F1 score in relation extraction

Combining LLMs with analytical methods shows potential for improvement

The dataset enables future research in mathematical relation extraction

Abstract

Recent advances in natural language processing (NLP), particularly with the emergence of large language models (LLMs), have significantly enhanced the field of textual analysis. However, while these developments have yielded substantial progress in analyzing natural language text, applying analysis to mathematical equations and their relationships within texts has produced mixed results. This paper takes the initial steps in expanding the problem of relation extraction towards understanding the dependency relationships between mathematical expressions in STEM articles. The authors construct the Mathematical Derivation Graphs Dataset (MDGD), sourced from a random sampling of the arXiv corpus, containing an analysis of $107$ published STEM manuscripts with over $2000$ manually labeled inter-equation dependency relationships, resulting in a new object referred to as a derivation graph that summarizes the mathematical content of the manuscript. The authors exhaustively evaluate analytical and machine learning (ML) based models to assess their capability to identify and extract the derivation relationships for each article and compare the results with the ground truth. The authors show that the best tested LLMs achieve $F_1$ scores of $\sim45\%-52\%$, and attempt to improve their performance by combining them with analytic algorithms and other methods.