PhysNLU: A Language Resource for Evaluating Natural Language Understanding and Explanation Coherence in Physics

TL;DR

PhysNLU introduces datasets to evaluate language models' ability to understand and generate coherent physics discourse, focusing on sentence order, relevance, and structure, revealing current model limitations.

Contribution

The paper provides new datasets and benchmarks for assessing physics-specific language understanding and coherence in models, highlighting their challenges in this domain.

Findings

Contemporary models struggle with physics discourse coherence.

Equations and sub-disciplines vary in frequency across physics texts.

Models perform poorly on sentence ordering and relevance tasks.

Abstract

In order for language models to aid physics research, they must first encode representations of mathematical and natural language discourse which lead to coherent explanations, with correct ordering and relevance of statements. We present a collection of datasets developed to evaluate the performance of language models in this regard, which measure capabilities with respect to sentence ordering, position, section prediction, and discourse coherence. Analysis of the data reveals equations and sub-disciplines which are most common in physics discourse, as well as the sentence-level frequency of equations and expressions. We present baselines that demonstrate how contemporary language models are challenged by coherence related tasks in physics, even when trained on mathematical natural language objectives.