Is More Context Always Better? Examining LLM Reasoning Capability for Time Interval Prediction

TL;DR

This study evaluates the ability of large language models to predict time intervals between user actions, revealing their limitations in capturing temporal structure and the nuanced effects of context size.

Contribution

It systematically benchmarks LLMs against statistical and machine learning models for temporal prediction, highlighting their current limitations and guiding future hybrid model design.

Findings

LLMs outperform simple statistical baselines but lag behind dedicated machine-learning models.

Adding more user context can decrease LLM prediction accuracy.

Moderate context improves LLM performance, but excessive detail degrades it.

Abstract

Large Language Models (LLMs) have demonstrated impressive capabilities in reasoning and prediction across different domains. Yet, their ability to infer temporal regularities from structured behavioral data remains underexplored. This paper presents a systematic study investigating whether LLMs can predict time intervals between recurring user actions, such as repeated purchases, and how different levels of contextual information shape their predictive behavior. Using a simple but representative repurchase scenario, we benchmark state-of-the-art LLMs in zero-shot settings against both statistical and machine-learning models. Two key findings emerge. First, while LLMs surpass lightweight statistical baselines, they consistently underperform dedicated machine-learning models, showing their limited ability to capture quantitative temporal structure. Second, although moderate context can improve LLM accuracy, adding further user-level detail degrades performance. These results challenge the assumption that "more context leads to better reasoning". Our study highlights fundamental limitations of today's LLMs in structured temporal inference and offers guidance for designing future context-aware hybrid models that integrate statistical precision with linguistic flexibility.