Does Time Have Its Place? Temporal Heads: Where Language Models Recall Time-specific Information

TL;DR

This paper identifies specific attention heads in language models that encode temporal information, showing their role in recalling time-specific facts and enabling targeted editing of temporal knowledge.

Contribution

The study introduces the concept of Temporal Heads, revealing their presence across models, their role in temporal fact recall, and methods for editing temporal knowledge.

Findings

Temporal Heads are present across multiple models.

Disabling these heads impairs time-specific recall.

Heads encode both numeric and textual temporal cues.

Abstract

While the ability of language models to elicit facts has been widely investigated, how they handle temporally changing facts remains underexplored. We discover Temporal Heads, specific attention heads that primarily handle temporal knowledge, through circuit analysis. We confirm that these heads are present across multiple models, though their specific locations may vary, and their responses differ depending on the type of knowledge and its corresponding years. Disabling these heads degrades the model's ability to recall time-specific knowledge while maintaining its general capabilities without compromising time-invariant and question-answering performances. Moreover, the heads are activated not only numeric conditions ("In 2004") but also textual aliases ("In the year ..."), indicating that they encode a temporal dimension beyond simple numerical representation. Furthermore, we expand the potential of our findings by demonstrating how temporal knowledge can be edited by adjusting the values of these heads.