Don't Fine-Tune, Decode: Syntax Error-Free Tool Use via Constrained Decoding

TL;DR

TOOLDEC is a constrained decoding algorithm that enforces syntax rules during language model output, significantly reducing errors and enabling out-of-the-box models to perform tool use as effectively as fine-tuned models.

Contribution

The paper introduces TOOLDEC, a novel decoding method using finite state machines to ensure syntax compliance, eliminating errors and improving tool use in large language models.

Findings

TOOLDEC eliminates all syntax errors in tool use tasks.

Applying TOOLDEC boosts out-of-the-box LLMs' tool use accuracy from 0% to 52%.

TOOLDEC achieves performance comparable to fine-tuned models without additional training.

Abstract

Instruction-tuned large language models (LLMs) excel at many tasks but often fail to use external tools due to complicated and unfamiliar syntax constraints. While extensive fine-tuning and prompting can mitigate the issue, these approaches are expensive and hard to generalize. Furthermore, because syntax constraints are only learned implicitly during fine-tuning, models still make frequent syntax errors. Motivated by the fact that these constraints can be better satisfied explicitly with constrained decoding, we propose TOOLDEC, a decoding algorithm using finite state machines to force LLMs to follow tool syntax. Our experiments show that TOOLDEC eliminates all syntax errors, achieving significantly better performance on various base models and benchmarks. More surprisingly, when applied to generalist out-of-the-box LLMs such as Mistral-Instruct, TOOLDEC improves its accuracy in tool use from the initial 0% to an impressive 52%, matching the performance of specialized fine-tuned models such as ToolLLM.