Falcon 7b for Software Mention Detection in Scholarly Documents

TL;DR

This study evaluates Falcon-7b's effectiveness in detecting and classifying software mentions in scholarly articles, emphasizing tailored training strategies to improve accuracy amidst complex academic language.

Contribution

It introduces a dual-classifier and adaptive sampling approach for software mention detection, providing insights into optimizing large language models for scholarly text analysis.

Findings

Selective labelling improves detection accuracy

Adaptive sampling enhances model performance

Combining multiple strategies yields limited additional benefits

Abstract

This paper aims to tackle the challenge posed by the increasing integration of software tools in research across various disciplines by investigating the application of Falcon-7b for the detection and classification of software mentions within scholarly texts. Specifically, the study focuses on solving Subtask I of the Software Mention Detection in Scholarly Publications (SOMD), which entails identifying and categorizing software mentions from academic literature. Through comprehensive experimentation, the paper explores different training strategies, including a dual-classifier approach, adaptive sampling, and weighted loss scaling, to enhance detection accuracy while overcoming the complexities of class imbalance and the nuanced syntax of scholarly writing. The findings highlight the benefits of selective labelling and adaptive sampling in improving the model's performance. However, they also indicate that integrating multiple strategies does not necessarily result in cumulative improvements. This research offers insights into the effective application of large language models for specific tasks such as SOMD, underlining the importance of tailored approaches to address the unique challenges presented by academic text analysis.