Transformers in the Dark: Navigating Unknown Search Spaces via Bandit Feedback
Jungtaek Kim, Thomas Zeng, Ziqian Lin, Minjae Lee, Chungpa Lee, Jy-yong Sohn, Hyung Il Koo, Kangwook Lee

TL;DR
This paper investigates whether Transformer-based models, including LLMs, can learn to perform search strategies in unknown tree-structured spaces using bandit feedback, aiming to improve problem-solving efficiency.
Contribution
The paper introduces a framework for evaluating Transformers' ability to approximate search algorithms in unknown tree spaces and demonstrates their capacity to learn and generalize search strategies.
Findings
Transformers can implement and learn search strategies from scratch.
Models generalize to longer horizons and deeper trees.
Fine-tuning LLMs on search trajectories enhances their search capabilities.
Abstract
Effective problem solving with Large Language Models (LLMs) can be enhanced when they are paired with external search algorithms. By viewing the space of diverse ideas and their follow-up possibilities as a tree structure, the search algorithm can navigate such a search space and guide the LLM toward better solutions more efficiently. While the search algorithm enables an effective balance between exploitation and exploration of a tree-structured space, the need for an external component can complicate the overall problem-solving process. We therefore pose the following question: Can LLMs or their underlying Transformer architectures approximate a search algorithm? To answer this question, we first introduce a simplified framework in which tree extensions and feedback signals are externally specified, allowing for controlled evaluation of search capabilities. We call this setting…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsArtificial Intelligence in Healthcare and Education · Multimodal Machine Learning Applications · Explainable Artificial Intelligence (XAI)
