Bilevel Optimization of Agent Skills via Monte Carlo Tree Search

TL;DR

This paper introduces a bilevel optimization framework using Monte Carlo Tree Search and large language models to systematically optimize agent skills, improving task performance.

Contribution

It formulates skill optimization as a coupled bilevel problem and employs MCTS and LLMs to jointly optimize skill structure and content.

Findings

Optimized skills lead to better agent performance on a question answering dataset.

The bilevel framework effectively balances structure and content optimization.

Experimental results show improved task accuracy with the proposed method.

Abstract

Agent \texttt{skills} are structured collections of instructions, tools, and supporting resources that help large language model (LLM) agents perform particular classes of tasks. Empirical evidence shows that the design of \texttt{skills} can materially affect agent task performance, yet systematically optimizing \texttt{skills} remains challenging. Since a \texttt{skill} comprises instructions, tools, and supporting resources in a structured way, optimizing it requires jointly determining both the structure of these components and the content each component contains. This gives rise to a complex decision space with strong interdependence across structure and components. We therefore represent these two coupled decisions as \texttt{skill} structure and component content, and formulate \texttt{skill} optimization as a bilevel optimization problem. We propose a bilevel optimization framework in which an outer loop employs Monte Carlo Tree Search to determine the \texttt{skill} structure, while an inner loop refines the component content within the structure selected by the outer loop. In both loops, we employ LLMs to assist the optimization procedure. We evaluate the proposed framework on an open-source Operations Research Question Answering dataset, and the experimental results suggest that the bilevel optimization framework improves the performance of the agents with the optimized \texttt{skill}.