Quality Matters: Evaluating Synthetic Data for Tool-Using LLMs

TL;DR

This paper emphasizes the importance of data quality in training tool-using LLMs, proposing two assessment methods and demonstrating that high-quality data enhances model performance even with less data.

Contribution

It introduces two novel approaches for evaluating synthetic data quality for training LLMs to use external tools, addressing a key gap in current research.

Findings

High-quality data improves model performance.

Model-driven assessment correlates with data reliability.

Quality checks are crucial for effective tool-using LLMs.

Abstract

Training large language models (LLMs) for external tool usage is a rapidly expanding field, with recent research focusing on generating synthetic data to address the shortage of available data. However, the absence of systematic data quality checks poses complications for properly training and testing models. To that end, we propose two approaches for assessing the reliability of data for training LLMs to use external tools. The first approach uses intuitive, human-defined correctness criteria. The second approach uses a model-driven assessment with in-context evaluation. We conduct a thorough evaluation of data quality on two popular benchmarks, followed by an extrinsic evaluation that showcases the impact of data quality on model performance. Our results demonstrate that models trained on high-quality data outperform those trained on unvalidated data, even when trained with a smaller quantity of data. These findings empirically support the significance of assessing and ensuring the reliability of training data for tool-using LLMs.