InSPO: Unlocking Intrinsic Self-Reflection for LLM Preference Optimization

TL;DR

InSPO introduces a self-reflective preference optimization method for LLMs that overcomes limitations of existing approaches by leveraging pairwise data and ensuring invariance to modeling choices, leading to more robust, human-aligned models.

Contribution

The paper proposes InSPO, a novel self-reflective preference optimization framework that improves alignment by utilizing pairwise responses and guaranteeing invariance to scalarization and reference choices.

Findings

InSPO outperforms DPO and RLHF in win rates.

It enhances response length control and robustness.

The method is plug-and-play with no extra inference cost.

Abstract

Direct Preference Optimization (DPO) and its variants have become standard for aligning Large Language Models due to their simplicity and offline stability. However, we identify two fundamental limitations. First, the optimal policy depends on arbitrary modeling choices (scalarization function, reference policy), yielding behavior reflecting parameterization artifacts rather than true preferences. Second, treating response generation in isolation fails to leverage comparative information in pairwise data, leaving the model's capacity for intrinsic self-reflection untapped. To address it, we propose Intrinsic Self-reflective Preference Optimization (InSPO), deriving a globally optimal policy conditioning on both context and alternative responses. We prove this formulation superior to DPO/RLHF while guaranteeing invariance to scalarization and reference choices. InSPO serves as a plug-and-play enhancement without architectural changes or inference overhead. Experiments demonstrate consistent improvements in win rates and length-controlled metrics, validating that unlocking self-reflection yields more robust, human-aligned LLMs. Our Code is available at https://github.com/Skylanding/InSPO.