Time-aware Prompting for Text Generation

TL;DR

This paper explores how incorporating timestamps into prompts enhances text generation, demonstrating that linear prompts improve overall quality and temporal consistency, while textual prompts better encode factual temporal information.

Contribution

It introduces two types of time-aware prompts for generation systems and a new dataset, TempWikiBio, for evaluating temporal effects in data-to-text tasks.

Findings

Linear prompts improve generation quality across datasets.

Textual prompts better encode factual temporal information.

Linear prompts are less sensitive to timestamp variations.

Abstract

In this paper, we study the effects of incorporating timestamps, such as document creation dates, into generation systems. Two types of time-aware prompts are investigated: (1) textual prompts that encode document timestamps in natural language sentences; and (2) linear prompts that convert timestamps into continuous vectors. To explore extrapolation to future data points, we further introduce a new data-to-text generation dataset, TempWikiBio, containing more than 4 millions of chronologically ordered revisions of biographical articles from English Wikipedia, each paired with structured personal profiles. Through data-to-text generation on TempWikiBio, text-to-text generation on the content transfer dataset, and summarization on XSum, we show that linear prompts on encoder and textual prompts improve the generation quality on all datasets. Despite having less performance drop when testing on data drawn from a later time, linear prompts focus more on non-temporal information and are less sensitive to the given timestamps, according to human evaluations and sensitivity analyses. Meanwhile, textual prompts establish the association between the given timestamps and the output dates, yielding more factual temporal information in the output.