TiMoE: Time-Aware Mixture of Language Experts
Robin Faro, Dongyang Fan, Tamar Alphaidze, Martin Jaggi

TL;DR
TiMoE introduces a time-aware mixture of language experts trained on disjoint time slices, enabling large language models to avoid temporal leakage and stay grounded in chronological knowledge while maintaining high performance.
Contribution
The paper proposes TiMoE, a modular, time-segmented pre-training method with causal routing, to improve temporal grounding in large language models.
Findings
Reduces future-knowledge errors by up to 15%
Matches or exceeds single-period expert performance
Effectively prevents temporal hallucinations
Abstract
Large language models (LLMs) are typically trained on fixed snapshots of the web, which means that their knowledge becomes stale and their predictions risk temporal leakage: relying on information that lies in the future relative to a query. We tackle this problem by pre-training from scratch a set of GPT-style experts on disjoint two-year slices of a 2013-2024 corpus and combining them through TiMoE, a Time-aware Mixture of Language Experts. At inference time, TiMoE masks all experts whose training window ends after the query timestamp and merges the remaining log-probabilities in a shared space, guaranteeing strict causal validity while retaining the breadth of multi-period knowledge. We also release TSQA, a 10k-question benchmark whose alternatives are explicitly labelled as past, future or irrelevant, allowing fine-grained measurement of temporal hallucinations. Experiments on eight…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsTopic Modeling · Artificial Intelligence in Healthcare and Education · Multimodal Machine Learning Applications
