ASPO: Adaptive Sentence-Level Preference Optimization for Fine-Grained Multimodal Reasoning

TL;DR

ASPO introduces a sentence-level preference optimization method that improves multimodal model alignment by providing fine-grained supervision, leading to better response quality without extra model complexity.

Contribution

This paper proposes ASPO, a novel adaptive sentence-level preference optimization technique that enhances multimodal model alignment by incorporating fine-grained supervision during training.

Findings

ASPO significantly improves multimodal model performance.

ASPO achieves better content assessment without extra parameters.

ASPO outperforms traditional binary preference optimization methods.

Abstract

Direct Preference Optimization (DPO) has gained significant attention for its simplicity and computational efficiency in aligning large language models (LLMs). Recent advancements have extended DPO to multimodal scenarios, achieving strong performance. However, traditional DPO relies on binary preference optimization, rewarding or penalizing entire responses without considering fine-grained segment correctness, leading to suboptimal solutions. The root of this issue lies in the absence of fine-grained supervision during the optimization process. To address this, we propose Adaptive Sentence-level Preference Optimization (ASPO), which evaluates individual sentences for more precise preference optimization. By dynamically calculating adaptive rewards at the sentence level based on model predictions, ASPO enhances response content assessment without additional models or parameters. This significantly improves the alignment of multimodal features. Extensive experiments show that ASPO substantially enhances the overall performance of multimodal models.