ACPO: Counteracting Likelihood Displacement in Vision-Language Alignment with Asymmetric Constraints

TL;DR

This paper introduces ACPO, a novel alignment method for vision-language models that prevents likelihood collapse and hallucinations by asymmetrically constraining preference optimization, leading to improved performance and reliability.

Contribution

ACPO is a new, modality-agnostic alignment technique that applies asymmetric scaling to preference optimization, effectively mitigating visual anchor collapse in multimodal models.

Findings

ACPO reverses likelihood degradation in models.

It reduces hallucinations on benchmark datasets.

ACPO improves overall model performance and stability.

Abstract

While Direct Preference Optimization (DPO) has become the de facto approach for aligning Large Vision-Language Models (LVLMs), it suffers from Likelihood Displacement, where the probability of both chosen and rejected responses collapses. This optimization flaw is especially detrimental in multimodal settings: the erosion of chosen likelihoods -- a failure we term Visual Anchor Collapse -- causes models to abandon visual evidence for strong language priors, precipitating significant hallucinations. To address this, we propose Asymmetric Constrained Preference Optimization (ACPO), a modality-agnostic alignment mechanism that applies dynamic, target-oriented scaling to preference optimization. ACPO derives a complexity-aware scaling coefficient applied exclusively to the rejected reward, asymmetrically suppressing the gradient flow on the rejected term while preserving the chosen distribution as a gradient-stable reference. While fundamentally a general-purpose objective, breaking this gradient symmetry is crucial for multimodal tasks, as it mitigates the suppression of visual tokens by language priors. Experiments on InternVL models demonstrate that ACPO effectively reverses the chosen-reward degradation of standard DPO. By halting Visual Anchor Collapse, ACPO generally outperforms baselines on hallucination benchmarks (HallusionBench, MM-IFEval) and general leaderboards (MMBench, MMStar, OCRBenchV2) while driving concurrent improvements in general capabilities.