Look It Up: Analysing Internal Web Search Capabilities of Modern LLMs

TL;DR

This paper evaluates how well modern large language models utilize integrated web search to improve factual accuracy, revealing strengths in static knowledge and challenges in dynamic, real-time information retrieval.

Contribution

Introduces a benchmark for assessing the necessity and effectiveness of web search in LLMs, highlighting current limitations and potential improvements.

Findings

Web access improves static accuracy for some models

Models often invoke search but with low accuracy on dynamic queries

Overconfidence and retrieval failures limit effectiveness

Abstract

Modern large language models integrate web search to provide real-time answers, yet it remains unclear whether they are efficiently calibrated to use search when it is actually needed. We introduce a benchmark evaluating both the necessity and effectiveness of web access across commercial models with no access to internal states or parameters. The dataset includes a static split of 783 temporally anchored questions answerable from pre-cutoff knowledge, aimed at testing whether models invoke search based on low internal confidence, and a dynamic split of 288 post-cutoff queries designed to test whether models recognise when search is required and retrieve updated information. Web access substantially improves static accuracy for GPT-5-mini and Claude Haiku 4.5, though confidence calibration worsens. On dynamic queries, both models frequently invoke search yet remain below 70 percent accuracy due to weak query formulation. Costs per accuracy-improving call remain low, but returns diminish once initial retrieval fails. Selective invocation helps, but models become overconfident and inconsistent after search. Overall, built-in web search meaningfully improves factual accuracy and can be invoked selectively, yet models remain overconfident, skip retrieval when it is essential, and falter once initial search queries underperform. Taken together, internal web search works better as a good low-latency verification layer than a reliable analytical tool, with clear room for improvement.