Time-Stamped Language Model: Teaching Language Models to Understand the Flow of Events

TL;DR

This paper introduces the Time-Stamped Language Model (TSLM) that enhances language models' ability to understand event sequences in procedural texts by incorporating timestamp encoding, leading to improved performance on related tasks.

Contribution

The paper proposes a novel timestamp encoding method within language models to better capture the flow of events in procedural texts, improving state-of-the-art results.

Findings

Achieved a 3.1% increase in F1 score on Propara dataset.

Outperformed previous models on location prediction in NPN-Cooking dataset.

Demonstrated general effectiveness for procedural text understanding.

Abstract

Tracking entities throughout a procedure described in a text is challenging due to the dynamic nature of the world described in the process. Firstly, we propose to formulate this task as a question answering problem. This enables us to use pre-trained transformer-based language models on other QA benchmarks by adapting those to the procedural text understanding. Secondly, since the transformer-based language models cannot encode the flow of events by themselves, we propose a Time-Stamped Language Model~(TSLM model) to encode event information in LMs architecture by introducing the timestamp encoding. Our model evaluated on the Propara dataset shows improvements on the published state-of-the-art results with a $3.1\%$ increase in F1 score. Moreover, our model yields better results on the location prediction task on the NPN-Cooking dataset. This result indicates that our approach is effective for procedural text understanding in general.