APEX: Learning Adaptive Priorities for Multi-Objective Alignment in Vision-Language Generation

TL;DR

APEX introduces a dynamic method for multi-objective alignment in vision-language models, balancing heterogeneous rewards and improving trade-offs across multiple objectives while maintaining stability.

Contribution

The paper presents APEX, a novel adaptive normalization and scheduling approach that addresses variance hijacking and gradient conflicts in multi-objective training.

Findings

Improved Pareto trade-offs on Stable Diffusion 3.5.

Balanced gains across multiple objectives.

Reduced instability in multi-objective alignment.

Abstract

Multi-objective alignment for text-to-image generation is commonly implemented via static linear scalarization, but fixed weights often fail under heterogeneous rewards, leading to optimization imbalance where models overfit high-variance, high-responsiveness objectives (e.g., OCR) while under-optimizing perceptual goals. We identify two mechanistic causes: variance hijacking, where reward dispersion induces implicit reweighting that dominates the normalized training signal, and gradient conflicts, where competing objectives produce opposing update directions and trigger seesaw-like oscillations. We propose APEX (Adaptive Priority-based Efficient X-objective Alignment), which stabilizes heterogeneous rewards with Dual-Stage Adaptive Normalization and dynamically schedules objectives via P^3 Adaptive Priorities that combine learning potential, conflict penalty, and progress need. On Stable Diffusion 3.5, APEX achieves improved Pareto trade-offs across four heterogeneous objectives, with balanced gains of +1.31 PickScore, +0.35 DeQA, and +0.53 Aesthetics while maintaining competitive OCR accuracy, mitigating the instability of multi-objective alignment.