InfoPO: On Mutual Information Maximization for Large Language Model Alignment

TL;DR

This paper introduces InfoPO, a new preference fine-tuning method for large language models that improves alignment with human preferences, especially in reasoning tasks, by avoiding overfitting and reliance on the Bradley-Terry model.

Contribution

The paper presents InfoPO, a novel preference fine-tuning algorithm that outperforms existing methods by eliminating the need for the BT model and enhancing reasoning capabilities.

Findings

InfoPO outperforms baselines on open benchmarks.

It improves reasoning task performance.

It prevents likelihood decrease during training.

Abstract

We study the post-training of large language models (LLMs) with human preference data. Recently, direct preference optimization and its variants have shown considerable promise in aligning language models, eliminating the need for reward models and online sampling. Despite these benefits, these methods rely on explicit assumptions about the Bradley-Terry (BT) model, which makes them prone to overfitting and results in suboptimal performance, particularly on reasoning-heavy tasks. To address these challenges, we propose a principled preference fine-tuning algorithm called InfoPO, which effectively and efficiently aligns large language models using preference data. InfoPO eliminates the reliance on the BT model and prevents the likelihood of the chosen response from decreasing. Extensive experiments confirm that InfoPO consistently outperforms established baselines on widely used open benchmarks, particularly in reasoning tasks.