Can Github issues be solved with Tree Of Thoughts?
Ricardo La Rosa, Corey Hulse, Bangdi Liu

TL;DR
This paper explores applying the Tree of Thoughts reasoning framework to improve LLMs in solving complex GitHub issues, but finds current methods insufficient without further enhancements.
Contribution
It introduces the application of the Tree of Thoughts framework to GitHub issue solving and analyzes its limitations for real-world problem-solving.
Findings
ToT alone does not outperform existing methods
Identifies need for deeper reasoning and agentic capabilities
Provides insights for future improvements in LLM reasoning
Abstract
While there have been extensive studies in code generation by large language models (LLM), where benchmarks like HumanEval have been surpassed with an impressive 96.3% success rate, these benchmarks predominantly judge a model's performance on basic function-level code generation and lack the critical thinking and concept of scope required of real-world scenarios such as solving GitHub issues. This research introduces the application of the Tree of Thoughts (ToT) language model reasoning framework for enhancing the decision-making and problem-solving abilities of LLMs for this complex task. Compared to traditional input-output (IO) prompting and Retrieval Augmented Generation (RAG) techniques, ToT is designed to improve performance by facilitating a structured exploration of multiple reasoning trajectories and enabling self-assessment of potential solutions. We experimentally deploy ToT…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsTopic Modeling · Explainable Artificial Intelligence (XAI) · Multimodal Machine Learning Applications
