Exploring Weaknesses in Function Call Models via Reinforcement Learning: An Adversarial Data Augmentation Approach

TL;DR

This paper introduces an adversarial data augmentation method using reinforcement learning to systematically identify and address weaknesses in function call capabilities of large language models, enhancing their robustness and generalization.

Contribution

The paper presents a novel RL-based adversarial training framework that targets specific weaknesses in function call models, improving their robustness beyond fixed-pattern data augmentation methods.

Findings

Enhanced robustness of function call models against adversarial queries

Systematic identification of model weaknesses through RL-generated adversarial data

Improved generalization of function call capabilities in LLMs

Abstract

Function call capabilities have become crucial for Large Language Models (LLMs), enabling them to interact more effectively with external tools and APIs. Existing methods for improving the function call capabilities of LLMs rely on data obtained either through manual annotation or automated generation by models, and use this data to finetune the LLMs. However, these methods often lack targeted design and are constrained by fixed patterns and data distributions, which limits their effectiveness in enhancing the generalization and robustness of function call LLMs. To address this limitation, we propose a novel adversarial data augmentation method that employs reinforcement learning to systematically identify and target the weaknesses of function call LLMs. Our training framework introduces a query model trained with reinforcement learning (RL) to generate adversarial queries that are specifically designed to challenge function call (FC) models. This approach adopts a zero sum game formulation, where the query model and the FC model engage in iterative alternating training. Overall, our method advances the development of more robust FC models and provides a systematic way to identify and correct weaknesses in the ability of LLMs to interact with external tools.