GeckOpt: LLM System Efficiency via Intent-Based Tool Selection

TL;DR

This paper proposes an intent-based tool selection method for LLMs that reduces token usage and costs by dynamically identifying user intent to streamline API calls, demonstrating promising efficiency improvements.

Contribution

It introduces a novel GPT-driven intent recognition approach to optimize tool selection, enhancing system efficiency in large-scale LLM deployments.

Findings

Token consumption reduced by up to 24.6%

Cost savings demonstrated on a large parallel platform

Potential for improved LLM system efficiency

Abstract

In this preliminary study, we investigate a GPT-driven intent-based reasoning approach to streamline tool selection for large language models (LLMs) aimed at system efficiency. By identifying the intent behind user prompts at runtime, we narrow down the API toolset required for task execution, reducing token consumption by up to 24.6\%. Early results on a real-world, massively parallel Copilot platform with over 100 GPT-4-Turbo nodes show cost reductions and potential towards improving LLM-based system efficiency.