Astro-NER -- Astronomy Named Entity Recognition: Is GPT a Good Domain Expert Annotator?

TL;DR

This paper explores using fine-tuned LLMs to assist non-experts in annotating astronomy literature for NER, aiming to address data scarcity and evaluate the quality of such collaborative annotations.

Contribution

It introduces a specialized annotation scheme for astronomy entities, creates a new dataset, and assesses the effectiveness of LLM-assisted annotation compared to expert labeling.

Findings

Moderate agreement between non-experts aided by LLM and domain experts.

Fair agreement between LLM predictions and domain experts.

The annotated dataset of 5,000 astronomy article titles is publicly available.

Abstract

In this study, we address one of the challenges of developing NER models for scholarly domains, namely the scarcity of suitable labeled data. We experiment with an approach using predictions from a fine-tuned LLM model to aid non-domain experts in annotating scientific entities within astronomy literature, with the goal of uncovering whether such a collaborative process can approximate domain expertise. Our results reveal moderate agreement between a domain expert and the LLM-assisted non-experts, as well as fair agreement between the domain expert and the LLM model's predictions. In an additional experiment, we compare the performance of finetuned and default LLMs on this task. We have also introduced a specialized scientific entity annotation scheme for astronomy, validated by a domain expert. Our approach adopts a scholarly research contribution-centric perspective, focusing exclusively on scientific entities relevant to the research theme. The resultant dataset, containing 5,000 annotated astronomy article titles, is made publicly available.