FedPDPO: Federated Personalized Direct Preference Optimization for Large Language Model Alignment

TL;DR

This paper introduces FedPDPO, a federated learning framework that personalizes large language model alignment with human preferences, effectively handling non-IID data and improving performance over existing methods.

Contribution

FedPDPO proposes a novel personalized federated preference optimization framework with parameter-efficient fine-tuning, addressing non-IID data challenges and enhancing LLM alignment with human preferences.

Findings

Achieves up to 4.80% accuracy improvement in federated settings.

Effectively handles non-IID preference data.

Demonstrates state-of-the-art performance on multiple datasets.

Abstract

Aligning large language models (LLMs) with human preferences in federated learning (FL) is challenging due to decentralized, privacy-sensitive, and highly non-IID preference data. Direct Preference Optimization (DPO) offers an efficient alternative to reinforcement learning with human feedback (RLHF), but its direct application in FL suffers from severe performance degradation under non-IID data and limited generalization of implicit rewards. To bridge this gap, we propose FedPDPO (Federated Personalized Direct Preference Optimization), a personalized federated framework for preference alignment of LLMs. It adopts a parameter-efficient fine-tuning architecture where each client maintains a frozen pretrained LLM backbone augmented with a Low-Rank Adaptation (LoRA) adapter, enabling communication-efficient aggregation. To address non-IID heterogeneity, we devise (1) the globally shared LoRA adapter with the personalized client-specific LLM head. Moreover, we introduce (2) a personalized DPO training strategy with a client-specific explicit reward head to complement implicit rewards and further alleviate non-IID heterogeneity, and (3) a bottleneck adapter to balance global and local features. We provide theoretical analysis establishing the probabilistic foundation and soundness. Extensive experiments on multiple preference datasets demonstrate state-of-the-art performance, achieving up to 4.80% average accuracy improvements in federated intra-domain and cross-domain settings.