ScoreFlow: Mastering LLM Agent Workflows via Score-based Preference Optimization

TL;DR

ScoreFlow introduces a gradient-based continuous optimization framework for large language model multi-agent workflows, significantly improving performance and scalability over existing discrete methods across various benchmarks.

Contribution

It presents ScoreFlow, a novel framework utilizing gradient-based optimization and a new preference method, Score-DPO, to enhance multi-agent LLM workflows with better flexibility and scalability.

Findings

Achieves 8.2% improvement over baselines on six benchmarks.

Enables smaller models to outperform larger ones with lower inference costs.

Demonstrates superior adaptability and scalability in complex problem-solving tasks.

Abstract

Recent research has leveraged large language model multi-agent systems for complex problem-solving while trying to reduce the manual effort required to build them, driving the development of automated agent workflow optimization methods. However, existing methods remain inflexible due to representational limitations, a lack of adaptability, and poor scalability when relying on discrete optimization techniques. We address these challenges with ScoreFlow, a simple yet high-performance framework that leverages efficient gradient-based optimization in a continuous space. ScoreFlow incorporates Score-DPO, a novel variant of the direct preference optimization method that accounts for quantitative feedback. Across six benchmarks spanning question answering, coding, and mathematical reasoning, ScoreFlow achieves an 8.2% improvement over existing baselines. Moreover, it empowers smaller models to outperform larger ones with lower inference costs. Project: https://github.com/Gen-Verse/ScoreFlow