LLM With Tools: A Survey

TL;DR

This survey explores how large language models can be integrated with external tools to improve their task performance, discussing methodologies, challenges, and innovative approaches including autonomous tool creation.

Contribution

It introduces a standardized paradigm for tool integration in LLMs, addressing key challenges and proposing techniques for better reasoning, diversity, and generalization.

Findings

Reproduced Chameleon's results on ScienceQA

Identified key challenges in tool invocation and selection

Explored techniques for fine-tuning and in-context learning

Abstract

The integration of tools in augmenting large language models presents a novel approach toward enhancing the efficiency and accuracy of these models in handling specific, complex tasks. This paper delves into the methodology,challenges, and developments in the realm of teaching LLMs to use external tools, thereby pushing the boundaries of their capabilities beyond pre-existing knowledge bases. We introduce a standardized paradigm for tool integration guided by a series of functions that map user instructions to actionable plans and their execution, emphasizing the significance of understanding user intent, tool selection, and dynamic plan adjustment. Our exploration reveals the various challenges encountered, such as tool invocation timing, selection accuracy, and the need for robust reasoning processes. In addressing these challenges, we investigate techniques within the context of fine-tuning and incontext learning paradigms, highlighting innovative approaches to ensure diversity, augment datasets, and improve generalization.Furthermore, we investigate a perspective on enabling LLMs to not only utilize but also autonomously create tools, which may redefine their role from mere tool users to tool creators. Finally,we reproduced Chameleon's results on ScienceQA and analyzed the code structure.