TReMu: Towards Neuro-Symbolic Temporal Reasoning for LLM-Agents with Memory in Multi-Session Dialogues

TL;DR

This paper introduces TReMu, a neuro-symbolic framework that enhances temporal reasoning in multi-session dialogues for LLM-agents by combining timeline summarization and code-based reasoning, supported by a new challenging benchmark.

Contribution

The paper presents a novel benchmark for temporal reasoning in multi-session dialogues and a neuro-symbolic framework that significantly improves LLMs' temporal reasoning capabilities.

Findings

Benchmark is challenging for existing LLMs.

Proposed framework raises reasoning accuracy from 29.83 to 77.67.

Neuro-symbolic approach outperforms baseline methods.

Abstract

Temporal reasoning in multi-session dialogues presents a significant challenge which has been under-studied in previous temporal reasoning benchmarks. To bridge this gap, we propose a new evaluation task for temporal reasoning in multi-session dialogues and introduce an approach to construct a new benchmark by augmenting dialogues from LoCoMo and creating multi-choice QAs. Furthermore, we present TReMu, a new framework aimed at enhancing the temporal reasoning capabilities of LLM-agents in this context. Specifically, the framework employs time-aware memorization through timeline summarization, generating retrievable memory by summarizing events in each dialogue session with their inferred dates. Additionally, we integrate neuro-symbolic temporal reasoning, where LLMs generate Python code to perform temporal calculations and select answers. Experimental evaluations on popular LLMs demonstrate that our benchmark is challenging, and the proposed framework significantly improves temporal reasoning performance compared to baseline methods, raising from 29.83 on GPT-4o via standard prompting to 77.67 via our approach and highlighting its effectiveness in addressing temporal reasoning in multi-session dialogues.