How Far are LLMs from Real Search? A Comprehensive Study on Efficiency, Completeness, and Inherent Capabilities

TL;DR

This paper investigates the potential of Large Language Models (LLMs) to perform effective search in problem-solving, introduces a framework to enhance search efficiency and completeness, and evaluates their capabilities across real-world tasks.

Contribution

The paper proposes Search via Learning (SeaL), a novel framework that leverages LLMs for efficient and complete search, and extends it to SeaL-C, demonstrating significant improvements in search accuracy and space reduction.

Findings

SeaL achieves near-perfect accuracy in planning tasks.

Search space is reduced by up to 99.1% using SeaL.

Current LLMs struggle with complex search tasks without systematic strategies.

Abstract

Search plays a fundamental role in problem-solving across various domains, with most real-world decision-making problems being solvable through systematic search. Drawing inspiration from recent discussions on search and learning, we systematically explore the complementary relationship between search and Large Language Models (LLMs) from three perspectives. First, we analyze how learning can enhance search efficiency and propose Search via Learning (SeaL), a framework that leverages LLMs for effective and efficient search. Second, we further extend SeaL to SeaL-C to ensure rigorous completeness during search. Our evaluation across three real-world planning tasks demonstrates that SeaL achieves near-perfect accuracy while reducing search spaces by up to 99.1% compared to traditional approaches. Finally, we explore how far LLMs are from real search by investigating whether they can develop search capabilities independently. Our analysis reveals that while current LLMs struggle with efficient search in complex problems, incorporating systematic search strategies significantly enhances their problem-solving capabilities. These findings not only validate the effectiveness of our approach but also highlight the need for improving LLMs' search abilities for real-world applications.