Teach model to answer questions after comprehending the document

TL;DR

This paper introduces a two-stage knowledge distillation approach for multi-choice machine reading comprehension, enhancing the model's ability to understand lengthy documents and improve answer accuracy.

Contribution

It proposes a novel two-stage knowledge distillation method that separates comprehension and answering, improving model understanding in MRC tasks.

Findings

Student model shows significant performance improvements.

Method effectively enhances comprehension of longer texts.

Two-stage approach outperforms single-stage models.

Abstract

Multi-choice Machine Reading Comprehension (MRC) is a challenging extension of Natural Language Processing (NLP) that requires the ability to comprehend the semantics and logical relationships between entities in a given text. The MRC task has traditionally been viewed as a process of answering questions based on the given text. This single-stage approach has often led the network to concentrate on generating the correct answer, potentially neglecting the comprehension of the text itself. As a result, many prevalent models have faced challenges in performing well on this task when dealing with longer texts. In this paper, we propose a two-stage knowledge distillation method that teaches the model to better comprehend the document by dividing the MRC task into two separate stages. Our experimental results show that the student model, when equipped with our method, achieves significant improvements, demonstrating the effectiveness of our method.