Encoder Fine-tuning with Stochastic Sampling Outperforms Open-weight GPT in Astronomy Knowledge Extraction

TL;DR

This paper introduces an encoder-based system using stochastic sampling and majority voting for extracting astronomy knowledge from research papers, outperforming open-weight GPT models.

Contribution

The paper presents a novel encoder fine-tuning approach with stochastic sampling and majority voting, improving knowledge extraction in astronomy literature.

Findings

Outperforms open-weight GPT baseline in astronomy knowledge extraction

Uses stochastic sampling and majority voting for fine-tuning

Simple, low-cost implementation with high accuracy

Abstract

Scientific literature in astronomy is rapidly expanding, making it increasingly important to automate the extraction of key entities and contextual information from research papers. In this paper, we present an encoder-based system for extracting knowledge from astronomy articles. Our objective is to develop models capable of classifying telescope references, detecting auxiliary semantic attributes, and recognizing instrument mentions from textual content. To this end, we implement a multi-task transformer-based system built upon the SciBERT model and fine-tuned for astronomy corpora classification. To carry out the fine-tuning, we stochastically sample segments from the training data and use majority voting over the test segments at inference time. Our system, despite its simplicity and low-cost implementation, significantly outperforms the open-weight GPT baseline.