Integrating External Tools with Large Language Models to Improve Accuracy

TL;DR

This paper introduces Athena, a framework that integrates external tools with large language models to significantly improve their accuracy in educational and reasoning tasks by accessing APIs and computational tools.

Contribution

The paper presents a novel framework for integrating external APIs and tools with LLMs, enhancing their reasoning accuracy in educational contexts.

Findings

Achieves 83% accuracy in mathematical reasoning

Achieves 88% accuracy in scientific reasoning

Outperforms state-of-the-art models like GPT-4o and LLaMA-Large

Abstract

This paper deals with improving querying large language models (LLMs). It is well-known that without relevant contextual information, LLMs can provide poor quality responses or tend to hallucinate. Several initiatives have proposed integrating LLMs with external tools to provide them with up-to-date data to improve accuracy. In this paper, we propose a framework to integrate external tools to enhance the capabilities of LLMs in answering queries in educational settings. Precisely, we develop a framework that allows accessing external APIs to request additional relevant information. Integrated tools can also provide computational capabilities such as calculators or calendars. The proposed framework has been evaluated using datasets from the Multi-Modal Language Understanding (MMLU) collection. The data consists of questions on mathematical and scientific reasoning. Results compared to state-of-the-art language models show that the proposed approach significantly improves performance. Our Athena framework achieves 83% accuracy in mathematical reasoning and 88% in scientific reasoning, substantially outperforming all tested models including GPT-4o, LLaMA-Large, Mistral-Large, Phi-Large, and GPT-3.5, with the best baseline model (LLaMA-Large) achieving only 67% and 79% respectively. These promising results open the way to creating complex computing ecosystems around LLMs to make their use more natural to support various tasks and activities.