Perhaps PTLMs Should Go to School -- A Task to Assess Open Book and Closed Book QA

TL;DR

This paper introduces a new task and benchmark for evaluating pre-trained language models on understanding instructional texts, highlighting their limited zero-shot and knowledge transfer capabilities in open and closed book question answering.

Contribution

It proposes a novel educational question-answering task with a new dataset and leaderboard, assessing PTLMs' ability to understand and utilize textbook content in different settings.

Findings

PTLMs perform around 50-56% on the task, close to random chance.

Adding textbooks to pre-training yields minimal performance gains.

Open-book setting improves accuracy to about 60%.

Abstract

Our goal is to deliver a new task and leaderboard to stimulate research on question answering and pre-trained language models (PTLMs) to understand a significant instructional document, e.g., an introductory college textbook or a manual. PTLMs have shown great success in many question-answering tasks, given significant supervised training, but much less so in zero-shot settings. We propose a new task that includes two college-level introductory texts in the social sciences (American Government 2e) and humanities (U.S. History), hundreds of true/false statements based on review questions written by the textbook authors, validation/development tests based on the first eight chapters of the textbooks, blind tests based on the remaining textbook chapters, and baseline results given state-of-the-art PTLMs. Since the questions are balanced, random performance should be ~50%. T5, fine-tuned with BoolQ achieves the same performance, suggesting that the textbook's content is not pre-represented in the PTLM. Taking the exam closed book, but having read the textbook (i.e., adding the textbook to T5's pre-training), yields at best minor improvement (56%), suggesting that the PTLM may not have "understood" the textbook (or perhaps misunderstood the questions). Performance is better (~60%) when the exam is taken open-book (i.e., allowing the machine to automatically retrieve a paragraph and use it to answer the question).