MapReduce LoRA: Advancing the Pareto Front in Multi-Preference Optimization for Generative Models

TL;DR

This paper introduces MapReduce LoRA and Reward-aware Token Embedding, novel methods that improve multi-preference alignment in generative models, achieving state-of-the-art results across text, image, and video tasks.

Contribution

The paper presents two new techniques for multi-preference optimization: parallel preference-specific training with iterative merging, and flexible preference control via token embeddings.

Findings

Significant improvements in image, video, and language generation metrics.

State-of-the-art multi-preference alignment performance across modalities.

Enhanced control and quality in generative models.

Abstract

Reinforcement learning from human feedback (RLHF) with reward models has advanced alignment of generative models to human aesthetic and perceptual preferences. However, jointly optimizing multiple rewards often incurs an alignment tax, improving one dimension while degrading others. To address this, we introduce two complementary methods: MapReduce LoRA and Reward-aware Token Embedding (RaTE). MapReduce LoRA trains preference-specific LoRA experts in parallel and iteratively merges them to refine a shared base model; RaTE learns reward-specific token embeddings that compose at inference for flexible preference control. Experiments on Text-to-Image generation (Stable Diffusion 3.5 Medium and FLUX.1-dev) show improvements of 36.1%, 4.6%, and 55.7%, and 32.7%, 4.3%, and 67.1% on GenEval, PickScore, and OCR, respectively. On Text-to-Video generation (HunyuanVideo), visual and motion quality improve by 48.1% and 90.0%, respectively. On the language task, Helpful Assistant, with Llama-2 7B, helpful and harmless improve by 43.4% and 136.7%, respectively. Our framework sets a new state-of-the-art multi-preference alignment recipe across modalities.