Temporal Alignment of Time Sensitive Facts with Activation Engineering

TL;DR

This paper introduces an activation engineering method to temporally align large language models, enhancing factual accuracy for specific time periods without additional training or datasets.

Contribution

The study presents a novel activation engineering approach for temporal alignment of LLMs, achieving comparable results to fine-tuning with less computational cost.

Findings

Up to 44% improvement in relative prompting accuracy

Up to 16% improvement in explicit prompting accuracy

Comparable performance to fine-tuning methods

Abstract

Large Language Models (LLMs) are trained on diverse and often conflicting knowledge spanning multiple domains and time periods. Some of this knowledge is only valid within specific temporal contexts, such as answering the question, "Who is the President of the United States in 2022?" Ensuring LLMs generate time appropriate responses is crucial for maintaining relevance and accuracy. In this work we explore activation engineering as a method for temporally aligning LLMs to improve factual recall without any training or dataset creation. In this research we explore an activation engineering technique to ground three versions of LLaMA 2 to specific points in time and examine the effects of varying injection layers and prompting strategies. Our experiments demonstrate up to a 44% and 16% improvement in relative and explicit prompting respectively, achieving comparable performance to the fine-tuning method proposed by Zhao et al. (2024) . Notably, our approach achieves similar results to the fine-tuning baseline while being significantly more computationally efficient and requiring no pre-aligned datasets.