Question-Answer Extraction from Scientific Articles Using Knowledge Graphs and Large Language Models

TL;DR

This paper introduces two methods for extracting question-answer pairs from scientific articles, one using large language models and the other leveraging knowledge graphs, with evaluations showing the effectiveness of the KG approach.

Contribution

The paper presents a novel combination of LLMs and knowledge graphs for extracting QA pairs from scientific literature, emphasizing the importance of fine-tuning ER models on scientific data.

Findings

KG-based QA extraction effectively captures main ideas

Fine-tuning ER models improves triplet quality

Subject matter experts validate the quality of generated QAs

Abstract

When deciding to read an article or incorporate it into their research, scholars often seek to quickly identify and understand its main ideas. In this paper, we aim to extract these key concepts and contributions from scientific articles in the form of Question and Answer (QA) pairs. We propose two distinct approaches for generating QAs. The first approach involves selecting salient paragraphs, using a Large Language Model (LLM) to generate questions, ranking these questions by the likelihood of obtaining meaningful answers, and subsequently generating answers. This method relies exclusively on the content of the articles. However, assessing an article's novelty typically requires comparison with the existing literature. Therefore, our second approach leverages a Knowledge Graph (KG) for QA generation. We construct a KG by fine-tuning an Entity Relationship (ER) extraction model on scientific articles and using it to build the graph. We then employ a salient triplet extraction method to select the most pertinent ERs per article, utilizing metrics such as the centrality of entities based on a triplet TF-IDF-like measure. This measure assesses the saliency of a triplet based on its importance within the article compared to its prevalence in the literature. For evaluation, we generate QAs using both approaches and have them assessed by Subject Matter Experts (SMEs) through a set of predefined metrics to evaluate the quality of both questions and answers. Our evaluations demonstrate that the KG-based approach effectively captures the main ideas discussed in the articles. Furthermore, our findings indicate that fine-tuning the ER extraction model on our scientific corpus is crucial for extracting high-quality triplets from such documents.