TUR-DPO: Topology- and Uncertainty-Aware Direct Preference Optimization

TL;DR

TUR-DPO enhances preference optimization for large language models by incorporating topology and uncertainty signals, improving alignment, faithfulness, and calibration without requiring reinforcement learning or online rollouts.

Contribution

It introduces a novel topology- and uncertainty-aware DPO variant that leverages reasoning topologies and calibrated uncertainty to improve model alignment and performance.

Findings

Improves judge win-rates, faithfulness, and calibration across multiple benchmarks.

Enhances performance in multimodal and long-context settings.

Matches or exceeds PPO on reasoning tasks while maintaining simplicity.

Abstract

Aligning large language models (LLMs) with human preferences is commonly done via reinforcement learning from human feedback (RLHF) with Proximal Policy Optimization (PPO) or, more simply, via Direct Preference Optimization (DPO). While DPO is stable and RL-free, it treats preferences as flat winner vs. loser signals and is sensitive to noisy or brittle preferences arising from fragile chains of thought. We propose TUR-DPO, a topology- and uncertainty-aware variant of DPO that rewards how answers are derived, not only what they say, by eliciting lightweight reasoning topologies and combining semantic faithfulness, utility, and topology quality into a calibrated uncertainty signal. A small learnable reward is factorized over these signals and incorporated into an uncertainty-weighted DPO objective that remains RL-free and relies only on a fixed or moving reference policy. Empirically, across open 7-8B models and benchmarks spanning mathematical reasoning, factual question answering, summarization, and helpful/harmless dialogue, TUR-DPO improves judge win-rates, faithfulness, and calibration relative to DPO while preserving training simplicity and avoiding online rollouts. We further observe consistent gains in multimodal and long-context settings, and show that TUR-DPO matches or exceeds PPO on reasoning-centric tasks while maintaining operational simplicity.