Maximizing mutual information between prompts and responses improve LLM personalization with no additional data or human oversight

TL;DR

This paper introduces MIPO, a contrastive data augmentation method that maximizes mutual information between prompts and responses, enabling LLMs to self-improve and personalize without external data or supervision.

Contribution

The paper proposes MIPO, a novel self-improvement framework that enhances LLM personalization and problem-solving by maximizing mutual information through contrastive data pairs.

Findings

MIPO achieves 3-40% improvements in personalized instruction-following.

MIPO yields 1-18% gains in math and multiple-choice tasks.

Maximizing mutual information enhances LLM self-improvement without extra data.

Abstract

While post-training has successfully improved large language models (LLMs) across a variety of domains, these gains heavily rely on human-labeled data or external verifiers. Existing data has already been exploited, and new high-quality data is expensive to collect. More fundamentally, true intelligence goes far beyond tasks that are easily verifiable. Therefore, we need self-improvement frameworks that allow models to improve without heavily relying on external oversight. We propose Mutual Information Preference Optimization (MIPO), a contrastive data augmentation method that constructs preference pairs by generating a positive response conditioning on the correct prompt, and a negative response by conditioning on a random, unrelated prompt. We show that using Direct Preference Optimization (DPO) to learn from this paired data maximizes pointwise conditional mutual information (MI), under the base LLM, between prompts and model responses. Empirical results with various-sized Llama- and Qwen-Instruct models show that when used to maximize MI between user context and response, MIPO provides an effective personalization technique, achieving 3-40% gains on personalized instruction-following compared to strong prompting baselines. Surprisingly, MIPO can also be applied to math and multiple-choice problem solving, yielding 1-18% gains without any additional data or human supervision. These results suggest a promising direction for self-improvement using intrinsic signals derived from contrastive data pairs.