Online Joint Fine-tuning of Multi-Agent Flows

TL;DR

This paper introduces an online joint fine-tuning method for multi-agent flows, enabling more efficient and effective construction of complex problem-solving architectures, demonstrated on a multi-hop QA dataset.

Contribution

It presents a novel online joint fine-tuning procedure for entire flows, inspired by Learning to Search, applicable to reward-free settings and leveraging simulator access.

Findings

Achieved state-of-the-art results on the Musique multi-hop QA dataset.

Demonstrated the effectiveness of joint fine-tuning over traditional stagewise methods.

Applicable to reward-free and simulator-based environments.

Abstract

A Flow is a collection of component models ("Agents") which constructs the solution to a complex problem via iterative communication. Flows have emerged as state of the art architectures for code generation, and are the raison d'etre for frameworks like Autogen. However, flows are currently constructed via a combination of manual prompt engineering and stagewise supervised learning techniques; the latter is limited to acyclic flows with granular node supervision. In this writeup I describe a procedure for online joint fine-tuning of an entire flow inspired by the Learning to Search framework. The approach leverages simulator access to reduce preferences over entire episodes to preferences over individual node outputs; when the components are language models the latter is a well-studied problem. The approach is applicable to reward-free settings (e.g., text feedback) if an episode evaluator model is available. I apply to the multi-hop QA dataset Musique achieving a state-of-the-art result.