Probing Script Knowledge from Pre-Trained Models

TL;DR

This paper investigates how well pre-trained language models like BERT understand script knowledge related to daily tasks, revealing strengths in temporal ordering but weaknesses in sub-event selection.

Contribution

It introduces three probing tasks to evaluate script knowledge in PLMs and analyzes BERT's capabilities in script induction and sub-event understanding.

Findings

BERT captures stereotypical temporal knowledge of sub-events

BERT poorly encodes inclusive and starting sub-event knowledge

Probing tasks can induce scripts from sub-events

Abstract

Script knowledge is critical for humans to understand the broad daily tasks and routine activities in the world. Recently researchers have explored the large-scale pre-trained language models (PLMs) to perform various script related tasks, such as story generation, temporal ordering of event, future event prediction and so on. However, it's still not well studied in terms of how well the PLMs capture the script knowledge. To answer this question, we design three probing tasks: inclusive sub-event selection, starting sub-event selection and temporal ordering to investigate the capabilities of PLMs with and without fine-tuning. The three probing tasks can be further used to automatically induce a script for each main event given all the possible sub-events. Taking BERT as a case study, by analyzing its performance on script induction as well as each individual probing task, we conclude that the stereotypical temporal knowledge among the sub-events is well captured in BERT, however the inclusive or starting sub-event knowledge is barely encoded.