Applying Relation Extraction and Graph Matching to Answering Multiple Choice Questions

TL;DR

This paper presents a method combining Transformer-based relation extraction with knowledge graph matching to answer multiple-choice questions accurately while maintaining process traceability, achieving around 70% correctness.

Contribution

It introduces a novel approach that dynamically generates and verifies knowledge graphs from text to improve MCQ answering accuracy and traceability.

Findings

Achieves up to 70% correct answers on MCQs.

Effectively verifies factual correctness of sentences using KGs.

Question category significantly impacts accuracy.

Abstract

In this research, we combine Transformer-based relation extraction with matching of knowledge graphs (KGs) and apply them to answering multiple-choice questions (MCQs) while maintaining the traceability of the output process. KGs are structured representations of factual knowledge consisting of entities and relations. Due to the high construction cost, they had been regarded as static databases with validated links. However, the recent development of Transformer-based relation extraction (RE) methods has enabled us to generate KGs dynamically by giving them natural language texts, and thereby opened the possibility for representing the meaning of the input sentences with the created KGs. Using this effect, we propose a method that answers MCQs in the "fill-in-the-blank" format, taking care of the point that RE methods generate KGs that represent false information if provided with factually incorrect texts. We measure the truthfulness of each question sentence by (i) converting the sentence into a relational graph using an RE method and (ii) verifying it against factually correct KGs under the closed-world assumption. The experimental results demonstrate that our method correctly answers up to around 70% of the questions, while providing traceability of the procedure. We also highlight that the question category has a vast influence on the accuracy.