Simulated Ignorance Fails: A Systematic Study of LLM Behaviors on Forecasting Problems Before Model Knowledge Cutoff

TL;DR

This study systematically evaluates whether simulated ignorance prompts can make large language models genuinely unaware of recent knowledge, revealing significant limitations and questioning the validity of retrospective forecasting benchmarks.

Contribution

First comprehensive analysis showing that simulated ignorance prompts do not reliably approximate true ignorance in large language models, highlighting methodological issues in forecasting evaluations.

Findings

SI leaves a 52% performance gap compared to TI

Chain-of-thought reasoning does not suppress prior knowledge

Models with better reasoning traces perform worse in SI fidelity

Abstract

Evaluating LLM forecasting capabilities is constrained by a fundamental tension: prospective evaluation offers methodological rigor but prohibitive latency, while retrospective forecasting (RF) -- evaluating on already-resolved events -- faces rapidly shrinking clean evaluation data as SOTA models possess increasingly recent knowledge cutoffs. Simulated Ignorance (SI), prompting models to suppress pre-cutoff knowledge, has emerged as a potential solution. We provide the first systematic test of whether SI can approximate True Ignorance (TI). Across 477 competition-level questions and 9 models, we find that SI fails systematically: (1) cutoff instructions leave a 52% performance gap between SI and TI; (2) chain-of-thought reasoning fails to suppress prior knowledge, even when reasoning traces contain no explicit post-cutoff references; (3) reasoning-optimized models exhibit worse SI fidelity despite superior reasoning trace quality. These findings demonstrate that prompts cannot reliably "rewind" model knowledge. We conclude that RF on pre-cutoff events is methodologically flawed; we recommend against using SI-based retrospective setups to benchmark forecasting capabilities.