Aligning Compound AI Systems via System-level DPO

TL;DR

This paper introduces SysDPO, a novel framework for aligning complex compound AI systems with human preferences by modeling them as DAGs and extending preference optimization techniques, addressing challenges of non-differentiability and system-level preference translation.

Contribution

The paper formulates compound AI systems as DAGs and develops SysDPO, a new method extending DPO for joint system-level alignment, with two variants for different data scenarios.

Findings

Effective joint alignment of language and diffusion models.

Successful alignment of LLM collaboration systems.

Demonstrated improvements in real-world applications.

Abstract

Compound AI systems, comprising multiple interacting components such as LLMs, foundation models, and external tools, have demonstrated remarkable improvements compared to single models in various tasks. To ensure their effective deployment in real-world applications, aligning these systems with human preferences is crucial. However, aligning the compound system via policy optimization, unlike the alignment of a single model, is challenging for two main reasons: (i) non-differentiable interactions between components make end-to-end gradient-based optimization method inapplicable, and (ii) system-level preferences cannot be directly transformed into component-level preferences. To address these challenges, we first formulate compound AI systems as Directed Acyclic Graphs (DAGs), explicitly modeling both component interactions and the associated data flows. Building on this formulation, we introduce $\textbf{SysDPO}$, a framework that extends Direct Preference Optimization (DPO) to enable joint system-level alignment. We propose two variants, SysDPO-Direct and SysDPO-Sampling, tailored for scenarios depending on whether we construct a system-specific preference dataset. We empirically demonstrate the effectiveness of our approach across two applications: the joint alignment of a language model and a diffusion model, and the joint alignment of an LLM collaboration system.