Manifold-Aware Temporal Domain Generalization for Large Language Models

TL;DR

This paper introduces MaT-LoRA, a parameter-efficient method that models temporal distribution shifts in large language models by constraining updates to a low-dimensional manifold, improving scalability and performance.

Contribution

It proposes a geometric reformulation of temporal domain generalization for LLMs using low-rank adaptation within a low-dimensional manifold, enabling scalable temporal modeling.

Findings

MaT-LoRA outperforms existing methods in temporal generalization tasks.

The approach maintains expressive power with reduced computational complexity.

Effective on diverse datasets including scientific, news, and review data.

Abstract

Temporal distribution shifts are pervasive in real-world deployments of Large Language Models (LLMs), where data evolves continuously over time. While Temporal Domain Generalization (TDG) seeks to model such structured evolution, existing approaches characterize model adaptation in the full parameter space. This formulation becomes computationally infeasible for modern LLMs. This paper introduces a geometric reformulation of TDG under parameter-efficient fine-tuning. We establish that the low-dimensional temporal structure underlying model evolution can be preserved under parameter-efficient reparameterization, enabling temporal modeling without operating in the ambient parameter space. Building on this principle, we propose Manifold-aware Temporal LoRA (MaT-LoRA), which constrains temporal updates to a shared low-dimensional manifold within a low-rank adaptation subspace, and models its evolution through a structured temporal core. This reparameterization dramatically reduces temporal modeling complexity while retaining expressive power. Extensive experiments on synthetic and real-world datasets, including scientific documents, news publishers, and review ratings, demonstrate that MaT-LoRA achieves superior temporal generalization performance with practical scalability for LLMs.