It's About Time: Incorporating Temporality in Retrieval Augmented Language Models

TL;DR

This paper introduces TempRALM, a temporally-aware retrieval augmented language model that improves the relevance of retrieved documents by considering both semantic and temporal factors, significantly enhancing answer accuracy for time-sensitive queries.

Contribution

The paper presents TempRALM, a novel approach that incorporates temporal awareness into retrieval augmented language models without additional pre-training or index modifications.

Findings

Up to 74% performance improvement over baseline RALM

Effective handling of temporal queries in information retrieval

No need for model re-training or index recalculation

Abstract

The web serves as a global repository of knowledge, used by billions of people to search for information. Ensuring that users receive the most relevant and up-to-date information, especially in the presence of multiple versions of web content from different time points remains a critical challenge for information retrieval. This challenge has recently been compounded by the increased use of question answering tools trained on Wikipedia or web content and powered by large language models (LLMs) which have been found to make up information (or hallucinate), and in addition have been shown to struggle with the temporal dimensions of information. Even Retriever Augmented Language Models (RALMs) which incorporate a document database to reduce LLM hallucination are unable to handle temporal queries correctly. This leads to instances where RALMs respond to queries such as "Who won the Wimbledon Championship?", by retrieving document passages related to Wimbledon but without the ability to differentiate between them based on how recent they are. In this paper, we propose and evaluate, TempRALM, a temporally-aware Retriever Augmented Language Model (RALM) with few-shot learning extensions, which takes into account both semantically and temporally relevant documents relative to a given query, rather than relying on semantic similarity alone. We show that our approach results in up to 74% improvement in performance over the baseline RALM model, without requiring model pre-training, recalculating or replacing the RALM document index, or adding other computationally intensive elements.