Agent Skill Acquisition for Large Language Models via CycleQD

TL;DR

CycleQD is a novel cyclic quality diversity approach that improves large language model skill acquisition by focusing on individual tasks, outperforming traditional fine-tuning and matching larger models' performance.

Contribution

Introduces CycleQD, a cyclic quality diversity method with model merging and SVD mutation, enhancing task-specific skill learning in large language models without data ratio tuning.

Findings

CycleQD surpasses traditional fine-tuning in coding, OS, and database tasks.

CycleQD achieves GPT-3.5-TURBO level performance with fewer parameters.

Method is applicable to image segmentation, demonstrating cross-domain versatility.

Abstract

Training large language models to acquire specific skills remains a challenging endeavor. Conventional training approaches often struggle with data distribution imbalances and inadequacies in objective functions that do not align well with task-specific performance. To address these challenges, we introduce CycleQD, a novel approach that leverages the Quality Diversity framework through a cyclic adaptation of the algorithm, along with a model merging based crossover and an SVD-based mutation. In CycleQD, each task's performance metric is alternated as the quality measure while the others serve as the behavioral characteristics. This cyclic focus on individual tasks allows for concentrated effort on one task at a time, eliminating the need for data ratio tuning and simplifying the design of the objective function. Empirical results from AgentBench indicate that applying CycleQD to LLAMA3-8B-INSTRUCT based models not only enables them to surpass traditional fine-tuning methods in coding, operating systems, and database tasks, but also achieves performance on par with GPT-3.5-TURBO, which potentially contains much more parameters, across these domains. Crucially, this enhanced performance is achieved while retaining robust language capabilities, as evidenced by its performance on widely adopted language benchmark tasks. We highlight the key design choices in CycleQD, detailing how these contribute to its effectiveness. Furthermore, our method is general and can be applied to image segmentation models, highlighting its applicability across different domains.